America Cancels 4,000 More Flights. FAA Denies Impact of New AI System. (kxan.com) 70
UPDATE: FAA Administrator Bryan Bedford said Saturday that U,S. airlines will likely cancel 4,000 to 5,000 of the roughly 41,000 flights scheduled for Saturday, reports Yahoo News. And as to the cause of the incident...
[U.S. Transportation Secretary Sean] Duffy said a router involved in the switch from copper wire to fiber "started flooding other routers in our air traffic control system." The system saw "over 1,000 connection attempts every four to five seconds," he said. That disrupted communications with airplanes and radar.
America's nationwide grounding of flights on Friday happened "amid growing scrutiny... of a new AI-powered air traffic management system" deployed by America's Federal Aviation Administration, reports Texas news station KXAN.
Ironically, that system is called SMART — but the outage happened "following warnings" from members of the U.S. Congress about this new AI system's safety, reliability and oversight: In an Oct. 13 letter to the Government Accountability Office, more than 20 members of Congress requested an independent review of the FAA's development and implementation of its Strategic Management of Airspace, Routes and Trajectories system, known as SMART. The lawmakers questioned whether the technology was being deployed with adequate input from frontline air traffic controllers, sufficient training and appropriate safety evaluations... In their letter to the GAO, the lawmakers specifically questioned whether the FAA had adequately evaluated risks associated with SMART's artificial intelligence capabilities, including the potential for the technology to interfere with experienced controllers' decision-making or generate inaccurate safety-critical information. "The nation's frontline aviation safety workforce apparently had little to no input into SMART's development and have received limited training and information during the current deployment process," the lawmakers wrote. The letter also questioned the FAA's decision to initially pilot the technology in heavily congested airspace, including the National Capital Region.
"Our switches, our routers, actually got overloaded with data," U.S. Transportation Secretary Sean Duffy acknowledged Friday.
But FAA Administrator Bryan Bedford said Saturday that officials have found "zero evidence of anything AI related" in Friday's outage, according to Yahoo News.
An article from The Hill.com notes that Duffy also said the new AI tool had nothing to do with the ground stop, and that he said the system "isn't plugged in." "This is completely disconnected, and I love that some people have said this would be the root cause. By the way, if it was, I'll tell you that, but this is not even plugged in," he said. "So, there's no possible way SMART would have impacted."
Duffy also asked Congress for $50 billion to rebuild what he described as an "incredibly old" air traffic control system.
Duffy had posted the day before that the AI system had been at least partially rolled out and that it was already working (adding "Soon this technology is going NATIONWIDE")
And Friday CNN's aviation analyst said they believed that AI system could be connected to the nationwide outage, calling it "a tremendous amount of coincidence," according to a report from Mediaite: CNN Aviation Analyst: "Many times in the past, what we have seen, and this has gone back decades, is that because air traffic control is this piecemeal, mishmash of a whole bunch of different systems... some going back to radio, telephones and floppy disks... when they put in new computer systems, they often have difficulty talking with, communicating with, or working with the much older systems. And I have to say, it's a tremendous amount of coincidence... If they're putting in new software now, this new SMART system to start managing that, and now we have a communications outage, I'm — we've seen this so many times in the past. I don't know if they're linked, but it's just too much coincidence that they were trying to start that moving today, and now we have problems with the en route system.
So if I was to guess — and we haven't heard any, you know, it's not that it's threats or criminals or terrorists. We haven't heard anything like that. So I am guessing that once again we have a problem with new air traffic control computer systems, AI — you know, modern systems trying to communicate with old systems. And we have just seen this so many times in the past, and it's a big problem... Our whole national airspace system is you know, pieces and parts. It's kind of a Franken-system. And that would be my guess, that they are having computer issues by putting the new software in with the old, just because today they announced that that were going to do this new system.
KXAN describes some of the complexity of the newly-launched AI system: According to the FAA, the system consolidates approximately 200 data streams, including airline schedules, flight paths, weather patterns, airport capacity, airspace conditions and air traffic controller staffing metrics. The agency says SMART uses artificial intelligence to analyze that information, predict traffic flows, identify potential conflicts and recommend adjustments to flight routes and departure schedules before disruptions occur. The FAA describes the technology as a shift from a reactive approach to a predictive one, allowing aviation officials to identify congestion, weather-related restrictions and other operational problems hours, days or even weeks in advance...
The congressional letter raised questions about whether the FAA's implementation strategy was moving faster than its safety evaluations, workforce training and independent oversight could support.
Ironically, the article points out that the FAA promoted anticipated benefits from the technology including fewer delays and flight cancellations. (Other anticipated benefits included improved on-time performance and faster recovery from weather disruptions.) Although the FAA has realized one of the system's anticipated benefits: reduced fuel consumption.
[U.S. Transportation Secretary Sean] Duffy said a router involved in the switch from copper wire to fiber "started flooding other routers in our air traffic control system." The system saw "over 1,000 connection attempts every four to five seconds," he said. That disrupted communications with airplanes and radar.
America's nationwide grounding of flights on Friday happened "amid growing scrutiny... of a new AI-powered air traffic management system" deployed by America's Federal Aviation Administration, reports Texas news station KXAN.
Ironically, that system is called SMART — but the outage happened "following warnings" from members of the U.S. Congress about this new AI system's safety, reliability and oversight: In an Oct. 13 letter to the Government Accountability Office, more than 20 members of Congress requested an independent review of the FAA's development and implementation of its Strategic Management of Airspace, Routes and Trajectories system, known as SMART. The lawmakers questioned whether the technology was being deployed with adequate input from frontline air traffic controllers, sufficient training and appropriate safety evaluations... In their letter to the GAO, the lawmakers specifically questioned whether the FAA had adequately evaluated risks associated with SMART's artificial intelligence capabilities, including the potential for the technology to interfere with experienced controllers' decision-making or generate inaccurate safety-critical information. "The nation's frontline aviation safety workforce apparently had little to no input into SMART's development and have received limited training and information during the current deployment process," the lawmakers wrote. The letter also questioned the FAA's decision to initially pilot the technology in heavily congested airspace, including the National Capital Region.
"Our switches, our routers, actually got overloaded with data," U.S. Transportation Secretary Sean Duffy acknowledged Friday.
But FAA Administrator Bryan Bedford said Saturday that officials have found "zero evidence of anything AI related" in Friday's outage, according to Yahoo News.
An article from The Hill.com notes that Duffy also said the new AI tool had nothing to do with the ground stop, and that he said the system "isn't plugged in." "This is completely disconnected, and I love that some people have said this would be the root cause. By the way, if it was, I'll tell you that, but this is not even plugged in," he said. "So, there's no possible way SMART would have impacted."
Duffy also asked Congress for $50 billion to rebuild what he described as an "incredibly old" air traffic control system.
Duffy had posted the day before that the AI system had been at least partially rolled out and that it was already working (adding "Soon this technology is going NATIONWIDE")
And Friday CNN's aviation analyst said they believed that AI system could be connected to the nationwide outage, calling it "a tremendous amount of coincidence," according to a report from Mediaite: CNN Aviation Analyst: "Many times in the past, what we have seen, and this has gone back decades, is that because air traffic control is this piecemeal, mishmash of a whole bunch of different systems... some going back to radio, telephones and floppy disks... when they put in new computer systems, they often have difficulty talking with, communicating with, or working with the much older systems. And I have to say, it's a tremendous amount of coincidence... If they're putting in new software now, this new SMART system to start managing that, and now we have a communications outage, I'm — we've seen this so many times in the past. I don't know if they're linked, but it's just too much coincidence that they were trying to start that moving today, and now we have problems with the en route system.
So if I was to guess — and we haven't heard any, you know, it's not that it's threats or criminals or terrorists. We haven't heard anything like that. So I am guessing that once again we have a problem with new air traffic control computer systems, AI — you know, modern systems trying to communicate with old systems. And we have just seen this so many times in the past, and it's a big problem... Our whole national airspace system is you know, pieces and parts. It's kind of a Franken-system. And that would be my guess, that they are having computer issues by putting the new software in with the old, just because today they announced that that were going to do this new system.
KXAN describes some of the complexity of the newly-launched AI system: According to the FAA, the system consolidates approximately 200 data streams, including airline schedules, flight paths, weather patterns, airport capacity, airspace conditions and air traffic controller staffing metrics. The agency says SMART uses artificial intelligence to analyze that information, predict traffic flows, identify potential conflicts and recommend adjustments to flight routes and departure schedules before disruptions occur. The FAA describes the technology as a shift from a reactive approach to a predictive one, allowing aviation officials to identify congestion, weather-related restrictions and other operational problems hours, days or even weeks in advance...
The congressional letter raised questions about whether the FAA's implementation strategy was moving faster than its safety evaluations, workforce training and independent oversight could support.
Ironically, the article points out that the FAA promoted anticipated benefits from the technology including fewer delays and flight cancellations. (Other anticipated benefits included improved on-time performance and faster recovery from weather disruptions.) Although the FAA has realized one of the system's anticipated benefits: reduced fuel consumption.
Yes and no (Score:2)
That was part of it, but the real issue is as always located inside management.
Re:Yes and no (Score:4, Funny)
Yes. The problem is that they called it AI, while it should have been called SI.
Re: Yes and no (Score:2)
SI is for Stupid Idiots.
Re: (Score:2)
Speaking of which, I have no idea who this Sean Duffy guy is (I'm not in the US), so I looked him up.
He's a former Fox TV presenter. The kakistoracy strikes again.
Re: (Score:2, Funny)
Super Human InTelligence
Re: Yes and no (Score:2)
Re: (Score:2)
Yes. The problem is that they called it AI, while it should have been called SI.
You joke, but in a press brief about the ground stop, DHS Secretary Markwayne Mullin corrected a reporter who asked about AI:
Markwayne Mullin Chides Reporter Amid FAA Ground Stoppage Chaos: ‘We Call it SI’ [mediaite.com]
“There are thoughts that, you know, this could be an AI related error or issue,” the reporter asked Mullin.
“We call it SI around here remember. SI,” he said, before adding, “You just can’t rule out anything at this point. We’re going through that process and we’re going to find out what happened.”
These renaming things are so pointless/stupid.
Re: (Score:2)
These renaming things are so pointless/stupid.
Stupid Indeed
Re:Yes and no (Score:5, Funny)
These renaming things are so pointless/stupid.
It's not stupid to them. They know that, if they don't do it, they'll have to deal with the wrath of their Stable Genius boss - and they'd like to keep their jobs.
Re: (Score:3)
These renaming things are so pointless/stupid.
These people renaming things are so pointless/stupid.
FTFY
Re: Yes and no (Score:2)
Re: (Score:2)
I think superintelligence has a well established meaning. It's the ENEMY of mankind. But why would the YOB care? His YUGE plan is to take the money, pardon himself (and all witnesses against him), run away, and then to boot it all he is probably sure that he'll be dead before any of his chickens try to come home. But jokes on the YOB if the malevolent superintelligence gets here before he can finish imploding/dying.
Re: Yes and no-EMP sensitivity? (Score:1)
Re: (Score:3)
Fiber is immune to EMP, except at the enter / exit nodes.
The claim is that it was a router problem (Score:5, Informative)
CBS reports [cbsnews.com] that:
The network DDoS'ed itself.
If the system is designed so that a couple of routers in one spot can make it necessary to ground everything, the design is bad. Saying they need $50B stat is wild when it clearly wasn't done right the first time with however much was spent so far.
Re:The claim is that it was a router problem (Score:5, Interesting)
The system is ancient and built for a time when floppy disks were common. Notably, they're still used to load weather data into the system.
Such systems do not expect modern volumes of data transport, nor are they built to handle it.
"But this fairly modern threat isn't handled well by this ancient system". Indeed. But updating such systems requires functioning bureaucracy, that isn't devoured by corruption on one side, and by ideology that views competence as the greatest evil that exists on the other.
It's in fact one of the greatest achievements of US that it can run with this level of dysfunction and remain clear #1 in the world in most productive aspects, including adoption of commercial aviation.
Re: (Score:2)
There are obviously design issues in the billion dollar system.
Re: (Score:2)
It may not have been thousands of simultaneous requests at the time the software was implemented, it may have been dozens or even fewer
Dont underestimate how much a legacy system or aspects of a legacy system can be overlooked when growth happens.
Re: (Score:2)
Re: (Score:1)
That weather data has now bloated from a few K to several M of JSON, repeated several times with different kinds of escaping, dozens of M of evilscript to process the JSON client-side to present the original Ks of data, and many M of CSS for because.
Re: (Score:2)
But did SMART generate the traffic? It's possible some bottleneck was the only thing which kept SMART stable and the fibre upgrade would have worked fine without it.
Re: (Score:2)
Why would it? It doesn't sound like it has anything to do with ATC, it's a recommendation system for scheduling.
Re: (Score:1)
The network DDoS'ed itself.
If the system is designed so that a couple of routers in one spot can make it necessary to ground everything, the design is bad. Saying they need $50B stat is wild when it clearly wasn't done right the first time with however much was spent so far.
I'm guessing someone could actually explain at a technical level exactly how these particular makes/models/configurations would DDoS "itself". Or better yet, have someone produce the actual audit trail of the events leading up to the network crashing, as every router is capable of logging.
You know, for shits and taxpayers sake. Make sure that cover story lines up with a $50B request. See who's asking for that pocket change.
Re: (Score:2)
actual audit trail of the events leading up to the network crashing
The mere thought of that audit occurring while this specific bunch of clowns is running the show, strikes me as insanely naive.
Re: (Score:2)
I hit this with broadcast ARP packets to move a few thous
Re: (Score:1)
If the system is designed so that a couple of routers in one spot can make it necessary to ground everything, the design is bad. Saying they need $50B stat is wild when it clearly wasn't done right the first time with however much was spent so far.
Networks are designed to route around damage but can't route around this Administration. :-)
Re:The claim is that it was a router problem (Score:5, Insightful)
For me, the bigger question is who the fuck authorized replacement of network equipment in the middle of the day? When I worked on an FAA contract, we had to perform even basic software updates overnight.
Indeed. If this really was a network issue, there is already a contract well under way to replace FTI with FENS. But if the story about the L3Harris contractor is true, then the problem was human error. If Mr. Duffy wants us to believe it takes $50 billion to fix human error during maintenance that shouldn't have been approved to be done during the day in the first place, then maybe he needs to spend more time in the Real World.
Re: (Score:2)
when it clearly wasn't done right the first time
You're making an assumption. Networks like this do not get designed and stay the same throughout their service life. They are built and the systems connected to them change over time. The problem isn't the network design wasn't done right, it is that it wasn't reviewed at every stage of a change in the systems on it.
L3Harris (Score:5, Insightful)
Re: (Score:2)
L3Harris has a long history of various contracting and implementation anomalies. As a large government contractor, with lots of moving parts and lots of contracts, that is not, itself, too surprising. However Duffy seems to be throwing them under the bus before the full facts in this case are known. That always raises an eyebrow or two.
Also, who does an IT infrastructure update/upgrade in the middle of a weekday, especially on an apparently critical piece?
Re: L3Harris (Score:1)
Network Issue? (Score:2)
On Saturday morning at their press conference (11am EST ish) the Transportation Secretary and FAA Administrators were blaming a routine remote site network upgrade (from copper to fiber) for the system instability.
Re: (Score:2, Funny)
Re: (Score:3, Insightful)
On Saturday morning at their press conference (11am EST ish) the Transportation Secretary and FAA Administrators were blaming a routine remote site network upgrade (from copper to fiber) for the system instability.
I posted this elsewhere, but it's worth noting again.
Who does an IT infrastructure update/upgrade in the middle of a weekday, especially on an apparently critical piece?
Re: (Score:2)
This and I would like to add. Who does an IT infrastructure update/upgrade this close to the busiest day(s) of the year for air travel? Maybe if you are building out a new infrastructure in parallel that is not interconnected yet but that does not seem to be the case.
What incredible reporting. (Score:2)
"And Friday CNN's aviation analyst said they believed that AI system could be connected to the nationwide outage, "
Wow, such dedicated investigative work. "Is it true that X?" "It could be. Dunno, really, but it could be connected."
Slashdot should be embarrassed for publishing this non-informative crap.
Re: (Score:2)
Slashdot should be embarrassed for publishing this non-informative crap.
They're probably proud instead, of posting sheerly speculative anti-AI FUD, because vibes say they should do that and be proud of it.
Re: (Score:2)
To add to the whole AI must succeed, we now have
Just remember (Score:5, Interesting)
Duffy's son-in-law's company got the contract for the FAA's AI tool after they'd made a 3 million donation, so of course Duffy has no reason to pull excuses out of his ass whatsoever...
Re: (Score:2)
On the other hand, no one has even alleged a mechanism for how the SMART system could control anything.
Re: (Score:2)
Duffy's son-in-law's company got the contract for the FAA's AI tool after they'd made a 3 million donation, so of course Duffy has no reason to pull excuses out of his ass whatsoever...
Adding... The PAC associated with the contractor, who was awarded the SMART system, donated to the congressional campaign of U.S. Rep.-elect Michael Alfonso (R-WI), who is Secretary of Transportation Sean Duffy's son-in-law. The contract it won was worth up to $875.9 million.
Google: FAA AI SMART Duffy son [google.com]
Startup eyed over flight chaos funnels cash to Transportation chief's son-in-law [rawstory.com]
Air Space Intelligence INC PAC, which is associated with the contractor Duffy announced in June would develop the Federal Aviation Administration's new AI-powered air traffic management system, contributed to the congressional campaign of U.S. Rep.-elect Michael Alfonso (R-WI)
Nothing to see here; move along... /s
Oh well ... (Score:2)
It appears that we took the training wheels off too soon.
SI not AI (Score:2)
It didn't scale? (Score:2)
To me it seems it had little to do with AI, just bad update/rollback procedures.
Re: It didn't scale? (Score:1)
Clearly had nothing to do with AI because if so they could've typed "Hey Claude, fix this mess. Make no mistakes".
That doesn't explain why Microsoft didn't use their AI to fix the Outlook outage. Weird
Right... (Score:2)
The explanation given at the initial press conference was such an obvious lie. It was a meaningless word salad. They blamed the outage on routers and switches. What? Seriously?
Do they understand that anyone with even the most basic knowledge of networking and/or IT would find that laughable?
Digital to analog failure (Score:2)
The explanation given at the initial press conference was such an obvious lie. It was a meaningless word salad.
Not the one I saw around 7pm Pacific. it sort of sounded like something was updated recently, integration testing apparently missed things, something else was overloaded with data and reboot, yet another thing did not restart properly, then it seemed some voice commands from air traffic control were not being broadcast, then they worried it might be happening all over the place, ... ground everything.
Basically the controller side has been digitized, the data then transmitted to an analog system for broad
Re: (Score:2)
Sure. It's not like a misconfigured router has ever taken down a network before.
lol. i was thinking about the fb debacle.. (Score:2)
Re:Right... (Score:4, Interesting)
I've actually been peripherally involved in something like this, I think it was September 2000, maybe a few months later.
An ATC system where they had two networks, the "office" network and the "core business of actually directing flights" network. The offices were around 10 miles away from the airport, which is where the ATC mainframe and - obviously- the ATC tower were. There was a bridge between the networks.
There were building works going on in the city, and they cut the cables while excavating. The agreement with the provider was that there would be two high speed lines and that they would not be in the same bundles of cables (or maybe glass fibre, if that was a "thing" back then). Unfortunately they had not insisted in the cables being in the separate ditches. The excavator cut both connections.
This should not have been a problem, two separate networks right? The critical one was at the airport and the whole point of that setup was so things would carry on if the offices were cut off. Most of the network group worked in the offices rather than at the airport. Some component was in the offices, the router at the airport was set up so that the control traffic to that component had the highest priority.
Bottom line, the core network was nominally unaffected, but a router there was trying to reach the office network and it was using all of its processor power for that. The core network pretty much came to a standstill. I can't remember how long that outage lasted but it was several hours.
Lessons were learned and all that.
The offices moved to a new location a year or so later, something which had been in preparation for a few years by then. New offices, new cables, whatever.
In 2003 (or maybe early 2004) something similar happened, an excavator chopped the connection. I was working on the mainframes at the airport from the offices so obviously I was affected, I just drove to the airport and carried on from there, everything worked exactly as it was supposed to - the offices could communicate with each other and the ATC Tower had full access to the ATC mainframes. It took a few hours before the two networks were linked to each other again but nobody outside noticed anything.
Re: (Score:2)
You've never seen a BGP problem take entire countries off the internet? Are you new here?
idiots (Score:1)
time to go after the criminals who are making these decisions
oh wait it's america. "let em cook!"
FUD Funding 101: Red Team (Score:1)
When a twenty-something strong network of nodes that controls air traffic handoff for every aircraft traveling between ATC towers (lets call it Achilles for DNS's sake) is targeted for a demonstration of just how weak our decades old system truly is regarding felling key dominos, it tends to scream one thing.
Red Team.
Duffy's Debrief was portraying the old man yelling at old systems so hard that he might as well been called Red Foreman.
C'mon folks. Zero risk to any American at any time, followed up by a
I like SMART air traffic control systems (Score:2)
Reflecting pool refurb reverb (Score:2)
Same idiots, same melody, slightly different song
Re: (Score:2)
1) Solicit political donaitions from cronies
2) Award government contracts based on who did and didn't donate
3) When work done by cronies' companies fails miserably, blame the Biden administration
4) Be sure to say "Thank You President Trump"
Project 2025 (Score:1)
They literally put it right in their documents you can read it yourself. I think they left the part out about giving all the contracts to their buddies but I mean honestly if you read that document and you didn't think that's what th
If you never brought down the network (Score:2)
Re: If you never brought down the network (Score:3)
Contrast with organizations with well defined CAPA and executive visibility into the process where everyone is religious about making sure It Never Happens Again
No! (Score:1)
Betteridge's law of headlines with a question mark says that these questions are always to be answered with 'NO'!
Sounds like blithering idiots at work (Score:2)
Competent people have a look what the network can take in addition to what it carries, then shape a new thing down well below that. And make sure they can turn it off fast. Sounds a like nobody even checked. Amateurs.
I have seen this extreme level of stupidity only with IBM tech consultants before. They killed the network of a large bank were I also transferred monitoring data, pretty much the same as they did. The difference is my first question was how much data and my second how much bandwidth can I safe
Trump on the FAA incident: (Score:1)
The below is what Trump is actually concerned about today according to his own official account:
“I am working with Senator Tim Scott and, believe it or not, Senator Adam Schiff and many others, both Republican and Democrat, on the Motion Picture, Television, and Entertainment Revitalization Act. The support and popularity of doing this, and bringing Entertainment back to the United States of America, has been unbelievable!
“Hollywood, and so many other places, have been virtually ‘emptied
Re: (Score:2)
Trump wasn't flying that day, so why should he care.
More so, why would he acknowledge a massive failure that happened on his watch.
Duffy complained that Democrats, former presidential administrations and previous sessions of Congress are to blame, ignoring the fact that Republicans have controlled Congress for 2+ years, the White House for 2 years and that only one Senator, Eric Schmitt (R-MO), has signed on to the Modern Skies Act. Seems a little self-serving and disingenuous. Plus the FAA already go
AI Pork pies (Score:1)
Just waiting for AI to cause mass casualties. (Score:2)
funding criteria (Score:2)
It turns out the primary determinant of whether a project can get funded is whether it is titled by a good acronym.
Update - new router went berserk (Score:2)