After 16 Years and $8 Billion, the Military's New GPS Software Still Doesn't Work (arstechnica.com) 73
An anonymous reader quotes a report from Ars Technica: Last year, just before the Fourth of July holiday, the US Space Force officially took ownership of a new operating system for the GPS navigation network, raising hopes that one of the military's most troubled space programs might finally bear fruit. The GPS Next-Generation Operational Control System, or OCX, is designed for command and control of the military's constellation of more than 30 GPS satellites. It consists of software to handle new signals and jam-resistant capabilities of the latest generation of GPS satellites, GPS III, which started launching in 2018. The ground segment also includes two master control stations and upgrades to ground monitoring stations around the world, among other hardware elements.
RTX Corporation, formerly known as Raytheon, won a Pentagon contract in 2010 to develop and deliver the control system. The program was supposed to be complete in 2016 at a cost of $3.7 billion. Today, the official cost for the ground system for the GPS III satellites stands at $7.6 billion. RTX is developing an OCX augmentation projected to cost more than $400 million to support a new series of GPS IIIF satellites set to begin launching next year, bringing the total effort to $8 billion.
Although RTX delivered OCX to the Space Force last July, the ground segment remains nonoperational. Nine months later, the Pentagon may soon call it quits on the program. Thomas Ainsworth, assistant secretary of the Air Force for space acquisition and integration, told Congress last week that OCX is still struggling. The GAO found the OCX program was undermined by "poor acquisition decisions and a slow recognition of development problems." By 2016, it had blown past cost and schedule targets badly enough to trigger a Pentagon review for possible cancellation.
Officials also pointed to cybersecurity software issues, a "persistently high software development defect rate," the government's lack of software expertise, and Raytheon's "poor systems engineering" practices. Even after the military restructured the program, it kept running into delays and overruns, with Ainsworth telling lawmakers, "It's a very stressing program" and adding, "We are still considering how to ensure we move forward."
RTX Corporation, formerly known as Raytheon, won a Pentagon contract in 2010 to develop and deliver the control system. The program was supposed to be complete in 2016 at a cost of $3.7 billion. Today, the official cost for the ground system for the GPS III satellites stands at $7.6 billion. RTX is developing an OCX augmentation projected to cost more than $400 million to support a new series of GPS IIIF satellites set to begin launching next year, bringing the total effort to $8 billion.
Although RTX delivered OCX to the Space Force last July, the ground segment remains nonoperational. Nine months later, the Pentagon may soon call it quits on the program. Thomas Ainsworth, assistant secretary of the Air Force for space acquisition and integration, told Congress last week that OCX is still struggling. The GAO found the OCX program was undermined by "poor acquisition decisions and a slow recognition of development problems." By 2016, it had blown past cost and schedule targets badly enough to trigger a Pentagon review for possible cancellation.
Officials also pointed to cybersecurity software issues, a "persistently high software development defect rate," the government's lack of software expertise, and Raytheon's "poor systems engineering" practices. Even after the military restructured the program, it kept running into delays and overruns, with Ainsworth telling lawmakers, "It's a very stressing program" and adding, "We are still considering how to ensure we move forward."
Oh but it works very well (Score:5, Insightful)
Re: (Score:1)
If only I had mod points!
Re: (Score:3)
This is so true, so true.
And it's not even US specific. In the wake of the Ukraine war, German parliament voted to give itself 100 billion of additional taxpayer money (i.e. debt) to spend on defense. Recently a report came out of all the money spent so far, 90% did not go towards the intended purpose.
Why any of the jokers in charge of our governments are still not in jail baffles me more and more every year. Oh yes, it's because they make the rules, sorry, my bad.
Re: Oh but it works very well (Score:1)
In California, taxes to fund road maintenance are the "for downloading Linux ISOs" argument for adding new forms of taxation, like the mileage tax that keeps getting proposed. Everybody here knows that will never get any funding no matter how many new taxes we add, but it makes a viable argument anyways because bad roads is something we all experience daily.
Re: (Score:1)
Why any of the jokers in charge of our governments are still not in jail baffles me more and more every year. Oh yes, it's because they make the rules, sorry, my bad.
No, it's because of all the idiotic enablers. We could just solve the problem by walking into the halls of power en masse and removing them but you can only get that kind of energy from total fucking clowns who want anarchy, and not the good kind that doesn't exist (as it leads naturally to feudalism) but the bad kind with only chaos.
Isn't that what AI is for? (Score:4, Funny)
Why don't they just ask Claude to fix it?
Better yet, give Elon Musk a couple hundred billion to start a private company and hire all the people, then hire that company to complete the project. Then it will be ready "this year".
Vibe code it! (Score:5, Funny)
Aerospace FFRDC role? (Score:5, Interesting)
Aerospace (and other FFRDCs like MITRE) exist to prevent massive failures like this. I wonder what the Aerospace corporate explanation is. I know from working at another FFRDC that often the worker-bees know the program is heading to disaster, but the managers won't carry the bad news to the customer. Other times, the bad news is delivered, but the government manager decides to carry on anyway. That can be due to pressure within the Service ("Don't f**k this up!, Colonel!"), or pressure from the contractor ("Trust us, these problems are temporary.")
The causes are often requirements instability, overly ambitious/unimplementable/unrealistic requirements, impossible initial schedule ("1 month to make the baby with 9 women"), technology problems (immature technology, vendors can't deliver as promised), and occasionally manufacturing/assembly/integration problems. And of course, substantial amounts of the functionality is in software, and this community knows the ways software projects can go south.
It's no consolation to this project,, its leadership, prime contractors, and customer community, but the last major project I worked on failed at 2 1/2 times the sunk cost of this one.
Re: (Score:2)
and TFA is a "there at it again..." type rage piece without giving even a taste of what many things really went wrong along the way from a design or other tech POV.
Re: (Score:2, Troll)
and TFA is a "there at it again..." type rage piece without giving even a taste of what many things really went wrong along the way from a design or other tech POV.
so? this should never happen.
whatever the reasons this is Category 5 Grade A incompetence.
the goddam Pentagon has FAILED 8 audits in a row.
i guess we can blame all that on DEI and everything will be rosy now that the Whites are back in charge.
Re:Aerospace FFRDC role? (Score:5, Insightful)
Failing audits is frankly independent from failing programs. The audits usually have problems tracking money flows and then property within the government. The contractor's expenditures are closely monitored. That doesn't mean they're in-line, but they're auditable. And when the audit discovers problems, there are ways for the government to respond. I've seen those applied rather frequently.
One common pattern is a program starts down the wrong path, and blows initial cost/schedule/performance. But that capability is needed badly (often because its predecessor program didn't deliver). So the Service piles on more requirements and 'readjusts the baseline' for additional funding, because "if we don't get it in this Program of Record, it'll be at least a decade before we can start a new Program of Record to get what we need." That just adds requirements to something that is already behind. If I had to guess what happened here, I bet there's some of that flavor over the execution. In my experience, most programs started with the combination of unachievable or under-specified requirements AND unachievable/unrealistic schedule.
(A 'Program of Record', by the way, consists of an approved requirements document, an approved POM budget for the next 5 years showing the RDTE money, the OPA purchasing money, and the OMA maintenance money FOR EACH FISCAL YEAR. If you run out of RDTE money but haven't finished the design, you're in trouble. The third element is the approved procurement strategy, that says how you'll buy it. That includes the kind of contract, firm fixed price or cost plus, the kinds of oversight, when and how prototypes will be delivered and tested, etc.)
Re: (Score:2)
and TFA is a "there at it again..." type rage piece without giving even a taste of what many things really went wrong along the way from a design or other tech POV.
so? this should never happen.
whatever the reasons this is Category 5 Grade A incompetence.
the goddam Pentagon has FAILED 8 audits in a row.
i guess we can blame all that on DEI and everything will be rosy now that the Whites are back in charge.
What part of "this" should never happen? I'm not a fan of massive contracts going over time and over budget... and this particular one seems to be an outlier, but the reality is that the majority of these large acquisition projects go over time and over budget and some of them outright fail. And it's been like this at least since "The Mythical Man Month" ( https://en.wikipedia.org/wiki/... [wikipedia.org] ) was first published 50+ years ago.
So just saying, as TFA does, "look at this project that's like 3x over duration
Re: (Score:2)
DoD is one of the largest organizations on the planet. Saying their failure to pass 8 audits in a row ignores the scale of the problem. First off, it was never built to be audited. The idea of auditing DoD is just like DoD telling a contractor to add this extra special whizzy to their weapons system after it had already been built. It has taken a lot of time and a lot of effort to corral systems that were never built with auditing in mind into sysrtems that can be audited. And those systems are not stand al
Re: (Score:2)
what you're describing is a system set up to foster corruption or be unable to contain it.
Re:Aerospace FFRDC role? (Score:4, Interesting)
If it's a cost-plus contact, the incentive is to milk it for as long as possible, without delivering a product. It's the definition of a perverse incentive.
Re: (Score:2)
I'd be interested to see how many times the customer came in with additional or changed requirements. I briefly did this kind of work and every time we got to a 60-75% solution, they'd come back with "oh, yeah, no, it has to do this now instead (or in addition)."
Then map the career path of the customer and see if they suddenly became a board member of the company.
Re: (Score:2)
Acquisition people have a 3 year block from working for their contractors. I don't know if the same rule applies to requirements people. For the large primes, that can entail a set of duties and a firewall between the part of the company that worked for the acquisition guy. Usually that works pretty well, but there are, of course, significant exceptions (and some people have gone to jail over that, see https://en.wikipedia.org/wiki/... [wikipedia.org] )
In defense of the 'revolving door': Companies really do need an un
Re: (Score:2)
Management won't tell the customerr, because they've agreed in insanely idiotic timeframes, and unachievable milestones.
I was employee 186, IIRC, of the startup division of Ameritech that was going to be your long distance company, in the mid-nineties.
Re: (Score:2)
The hope is to lock in the contract with totally unrealistic cost/schedule, and then make up the financial risks through Engineering Change Proposals. Those ECPs reflect both requirements instability and cost/schedule "rebaselining" instability. I think managing ECPs is what separates the really good Program Managers/System Program Officers from the merely competent PM/SPO.
Re: (Score:2)
and, sadly, bidding this way is probably rational. If you did it more realistically you'd be at 2x the price of other bids and wouldn't get it.
it's like... ecology or selection... this is the "organism" that's apparently fit for the environment. which I didn't mean that it shouldn't be improved... but I'm not so eager to label the participants evil or dumb... (I mean I AM , but try to keep it in check through such reframings)
Re: (Score:2)
That's the challenge for the source selection/proposal evaluation. It's tough to write independent criteria that allow you to throw out a bid as "implausible". (I remember one where the total travel budget for the 3 year period of performance was 2 people, 1 week, 1 trip per year. This was on the same project where I was on 75% travel. I asked, 'How do we mark this as bullshit?' and was told, "Well, that goes into the overall evaluation criterion for 'cost realism'. That group won the bid, by the way.)
Re: Aerospace FFRDC role? (Score:1)
It's called corruption (Score:5, Insightful)
Eisenhower knew the deal in 1961.
Re:It's called corruption (Score:4, Insightful)
Re:It's called corruption (Score:5, Insightful)
Reagan was on a privatization kick. It resulted in wonderful growth for the Beltway Bandits. Another reason privatization does not work for government is because government is not private enterprise that can decide what markets to enter and which to exit. Programs are mandated by law. Sure laws can change but it is a long arduous path. And you wouldn't want it any other way. Changing things on a whim has brought current U.S. to its knees, and the damage appears to be long lasting.
Want to see privatization at work? Look at the U.S. health system. Those insurance companies use actuaries to determine who gets covered. A good team of actuaries can put a price on your grandmother and her cat and will if you ask them to. As a consequence, we have a health system which can send you to the poor house in under a year because of a medical condition.
Re: (Score:1)
If government did produce, it would quickly crowd out the private sector until there was nothing left but a totalitarian dictatorship.
Re:It's called corruption (Score:4, Insightful)
Sometimes yes. But it's not usually the government itself producing and thus pushing out private enterprise. Usually it's the leader establishing close personal relationships with loyal private businesses, and funneling money their way without bidding or due process. A king picking the winners and losers. This is what Hitler did in the 30s and 40s as he built up his war machine, what the CCP in china does now, and what Trump is doing in the USA right now. In fact Brandon Carr just boasted in front of an audience of thousands how he has used the power of government to get rid of Trump's media critics and helped his friends buy up all the big media companies.
Re: (Score:1)
What I'm talking about is the public sector crowding out the private until there isn't enough of the private economy left to operate a free market. You can't
Re: (Score:3)
Go fuck yourself. And enjoy it when you need medical care, and your insurance company refuses you and then drops you.
I also do not approve of my tax dollars being wasted on scum like you.
My last day job, before I retired, was as sr. Linux admin at the NIH for t0 years. Worked for a contractor. Did the *same* job those 10 years. Looked it up, had about the same salary and benefits that a fed would. Oh, but because privatize, our tax dollars was paying for a fed to waste their time managing the contracts. And
Re: (Score:2)
Telling me to fornicate with myself and following it with a non-sequitur. Brilliant. I bow to your intellect.
Re: (Score:2)
Private enterprises are not productive enterprises; it is a necessary cost of society. Private enterprises do not build anything and possesses no means to do so. People do so, and so private enterprise must pay people for what they need.
Re: (Score:2)
I wish it was corruption - it's bad management (Score:5, Interesting)
Then there is the management of projects. Now the de facto person in charge is the one in charge of the Jira tickets. They decide what tasks get resources and they decide which engineers do what tasks. They can't understand a development plan, they can't build the code, they usually can't even use the product. What they can do is take bugs from testing, create tickets and assign engineers to the tickets. Their only metric is how fast tickets are resolved. They don't care if the engineer they assign knows nothing about the project
The most successful embedded engineers today
Don't put useful comments their code (they may describe how the code works but the code says what it does)
They never document (and most projects don't even have a single place to put documents or have a way to find them)
They are good at volunteering for easy or high visible tickets
They close bugs by creating global variables that track the condition of the bug, then adding a function that is called all the time, a function that then checks the globals for the error condition, prevents (or masks) the error and then (hopefully) cleans up the globals without creating too many other bugs.
- This means they never have to understand the code. If there is automated testing they might never even need to know how to use the product.
Re: (Score:3)
Is the rise of cheap but powerful embedded processors and the Arduino ecosystem partly to blame? I don't think I've seen very much high-quality and efficient arduino programming done. C++ with lots of unnecessary OOP. As long as there's a watchdog timer to reboot the thing every couple of hours, she'll be right. Ship it. Heck I use micropython for a lot of projects lately. It's very cool but it may not be preparing young developers for the rigors of aerospace development.
Re: (Score:3)
Recommended reading (Score:5, Insightful)
"The Spoils of War" by Andrew Cockburn. Goes way back to American soldiers having to steal boots off dead Chinese soldiers in Korea to get decent boots, their feet were freezing off.
"The Pentagon Wars" by Col. James F Burton. Burton was part of the 1980s "Fighter Mafia" who got the F-16 built, against Pentagon tendencies for every new plane to be twice the weight and twice the cost of the last one. (The F-35 continues the tradition.) They were the ones who publicized the $400 hammer and $600 toilet seat.
Cockburn laments that people thought it only applied to some things, when their point was that every $1 lightbulb on the console was $25 to replace.
Burton notes that one Army logistics guy got the price of a single uranium bullet down from $80 to $4 by whipsawing two suppliers into real competition, another reduction every purchasing round, for years. That guy gave a presentation to a roomful of Stars on it, and came back to his desk to find retirement papers waiting. Or a transfer to Thule. His choice.
Re: (Score:2)
Re: (Score:2)
Those "Stars" that fired him need to have their arses handed to them. Corruption is something I fear we will never be able to solve, but I live in eternal hope.
Yea, but as long as the size of your budget determines how much staff you get and how important you are, no.
Re:Recommended reading (Score:5, Insightful)
"The Pentagon Wars" by Col. James F Burton. Burton was part of the 1980s "Fighter Mafia" who got the F-16 built, against Pentagon tendencies for every new plane to be twice the weight and twice the cost of the last one. (The F-35 continues the tradition.)
John Boyd was part of those battles, recounted in Comer's excellent biography of him, titled "Boyd: The Fighter Pilot Who Changed the Art of War."
They were the ones who publicized the $400 hammer and $600 toilet seat.
While I like those stories and many pointed out serious problems in government procurement, some also fail to tell the whole story because the headline is what someone wants since it causes outrage.
We used bolts that cost a lot of money and looked the same as a 10 cent one from a hardware store, but ours were designed to perform to an exact spec, traceable to the ore, and tested to ensure they meet specs. You don't want that bolt to fail to performa at a critical moment when you are above, on, or below the ocean. We had bronze tools that cost a lot but looked like ordinary tools because sparks around oxygen tanks can cause issues. My point: There is a lot of wastefull spending and overpriced stuff in government contracting, but sometimes there is more to the story than simply "Military spends $x00 for something that is $10 at Walmart..."
One big problem is how the government budgets. You generally have to spend all you money on an annual basis, and if you give money back that you saved, you risked getting less next year because "you didn't need as much as you said last year..." So, come Q4, you go on a spending spree to spend whatever is left; trying to spend more is better than trying to spend less.
Re: (Score:2)
Burton was part of the 1980s "Fighter Mafia" who got the F-16 built, against Pentagon tendencies for every new plane to be twice the weight and twice the cost of the last one.
Eh, the "Fighter Mafia" may have had some influence on the development of the F-16, but if anyone had actually listened to them we would have had a fighter with no radar, a minimal missile loadout, and no air-to-ground capabilities. It would definitely not be the workhorse of air forces that the F-16 still is today.
Re: Recommended reading (Score:2)
Thatâ(TM)s exactly what it was. The YF-16 which won the contract had no radar and minimal loadout. It was the result of the work the Fighter Mafia put in, reinventing what a jet fighter was.
That was compromised slightly when the FSD order was placed, and the first delivered F-16 was some 25% heavier than the YF-16. But the core plane to which the radar was added was not merely influenced by the Fighter Mafia, but built entirely on their concepts. Thatâ(TM)s how it has remained competitive for so l
Re: (Score:2)
"The Spoils of War" by Andrew Cockburn. Goes way back to American soldiers having to steal boots off dead Chinese soldiers in Korea to get decent boots, their feet were freezing off.
Still probably true BTW, though ironically my example is the opposite. I bought a pair of issue desert boots and a paid of issue arctic boots at the same time. The arctic boots came with two sets of liners, were flawless, I still have them. The desert boots came apart on like the second wear, which sadly was long after I bought them. They disintegrated at a seam.
This How To Fix The Problem (Score:5, Funny)
Re:This How To Fix The Problem (Score:4, Interesting)
Turn the project over to the Ukrainian they'd get it fixed so they could use the full military precision version of our GPS for themselves.
This is likely way to true and they'd do for 5% of the cost.
Re: (Score:1)
I would like to say how cool your Google+ login is. It shows up in your post like a bright red beacon. I once even wrote a haiku-poem for Google+ login icons on Slashdot. Take it! It is yours!
Google Plus Login
red on a green Slashdot sea
setting my soul free
exotic matter
how I long to fly with you
Google Plus Login
Google Plus Login
fancy Google Plus Login
Google Plus Login
primitive tribesmen
gaze at the little red square
dream of things to come
I could do it cheaper (Score:5, Funny)
I'm not a programmer, but I could make a system that doesn't work for only $4B. I'm only discounting it because I want to save taxpayer money.
Re: (Score:2)
Where is DOGE now? (Score:5, Insightful)
Subject is the question. Where is DOGE on the big stuff? The Pentagon wastes more every month in fraud, waster, and abuse than USAID spends annually. But somehow charity gets the axe and Ratheon keeps getting multi-billion dollar contracts, no strings attached. Can anyone put aside the woke distraction and look at the serious problems?!?
Re: (Score:2)
Re: (Score:1)
Re: (Score:2)
Re: (Score:3)
The cart before the horse (Score:2)
Re: (Score:2)
Re: (Score:2)
They have working software: OCX is intended to replace the current ground system, Operational Control System (OCS). They have launched a lot more than 30 satellites -- in fact, most of them have been decommissioned, although the currently operational set have mostly outlived their design lifetimes by a lot. The oldest active satellites are Block IIR satellites, with a design life of 7.5 years ... and launched between 1997 and 2004 (so the youngest of them would be old enough to drink alcohol in the US).
ht [wikipedia.org]
Re: (Score:2)
Wrong programming team (Score:2)
They should have hired Ukrainian programmers.
Behind schedule, over budget (Score:2)
Re: (Score:2)
Not Surprised (Score:5, Insightful)
RTX Corporation, formerly known as Raytheon, hasn't produced a single thing of halfway decent quality in many decades and only hires the shittiest engineers.
Re: (Score:3)
Re: (Score:3)
Maybe they need motivation (Score:2)
Why not contract out program development to some of those Iranian hackers who seem to have drones flying at will over US military bases? It might not be a perfect solution, but it would probably be cheaper and better than the clusterf^ck they've had going on for the best part of a decade.
ADA (Score:2)
Just curious, do they develop it with ADA?
Never mind military doctrine dictates more (Score:2)
Bungled from the beginning (Score:3)
Wha? Another military boondoggle? (Score:1)
Itâ(TM)s almost like everything the weapons companies do is designed to maximize profit for the longest period of time while producing substandard results!
Business Plan (Score:2)
The next Pentagon software contract that comes up, NIX Inc. (TM) will offer to do nothing for only 15 years and for just $7 billion.
Profit !