Data-Sifting For Timely Intelligence Still an Elusive Goal 131
gyrogeerloose writes "Although there was evidence to suggest that the Japanese navy was up to something in December 1941, that information was scant and came too late. Today's intelligence agencies have another problem altogether — more information than they can deal with, and computers aren't helping as much as one might expect for reasons that will be familiar to Slashdot readers: computers can crunch numbers faster and more accurately than humans, but they're still easily baffled by language as it is commonly used in the real world. Metaphor, slang and simple figures of speech can confuse the best algorithm and, as quoted in the linked article in the San Diego Union-Tribune, 'A system that takes a week to discover a bombing that will occur in a day isn't very useful.'"
This stuff is hard. I do it every day... (Score:4, Interesting)
Computer Analysis Feedback Loop (Score:1, Interesting)
"A system that takes a week to discover a bombing will occur in a day isn't very useful.'"
I would argue that it can be. If your system takes a week to discover a bomb plot that occurred last week, refinements can be made to the system to improve its detection abilities.
Any statistician could have told them that (Score:3, Interesting)
There's one thing that's worse than too little data: Too much data that may or may not be relevant to your task.
It's bad to have no data. But that can be remedied. Having more data than you can process, worse, data where you cannot discriminate between wheat and chaff is pretty much useless. And that's basically what we have now. They were busy collecting data left and right, not asking whether that data could be relevant. Now they're stuck with a buttload of data that may or may not be relevant.
The best solution? Toss it and start over. And this time, collect only what's relevant.
Re:HUMINT SIGINT (Score:4, Interesting)
Which is why human intelligence is much more useful than signal intelligence (data mining).
Exactly. If people can't sift through the mass of information (and misinformation) we have today, what hope does a computer have? Just look at how hard it is to find "The Truth" in todays news, or on the Internet...
Re:HUMINT SIGINT (Score:3, Interesting)
Which is why human intelligence is much more useful than signal intelligence
People lie.
The US government is especially good at sending bogus signals. There's no reason to believe other governments aren't as good.
All intelligence has it's problems. The trick is to put together enough different sources to weed out the bogus, and home in on the truth, all while keeping everything secret. Basically, it's an impossible task, but sometimes it's good enough, and sometimes you go to war looking for WMD that aren't there.
Big Brother is watching you (Score:4, Interesting)
Re:America forced Japan's hand (Score:4, Interesting)
It was strategically the right move.
I think the outcome of World War II demonstrates that it was not the right move strategically. Tactically, perhaps, if the Japanese military planners were expecting the U.S. to enter into war, but it was a strategic disaster for them over the long run.
Re:HUMINT SIGINT (Score:1, Interesting)
Oh shut up already. If a country X decides to threaten another country with nukes, EVERY other country that has nukes will be very displeased and tell country X to GTFO. The whole reason that nukes are great as a defense is that nobody wants to be involved in a nuclear war. Which is also why they suck for offense. The only reason we don't want some countries to have nukes is because it sucks when those things get lost/stolen/whatever. A country is unable to use nukes, but a person or an organisation is.
We're *not* afraid that Iran will declare a war on someone and threaten to use nukes, because that would be suicide for Iran. Having said that, one might suspect that if Iran were to be in possession of such weapons, those bombs might find their way into other peoples hands (on purpose or by accident).
From Irans perspective, I can imagine wanting some big bombs to discourage an American invasion (the USA seems to like invading countries in the Middle East, and a good reason for an invasion in optional...).
Or maybe Iran simply wants nuclear power and the rest of the world should just gtfo and mind their own business.
Re:The Real Issue (Score:5, Interesting)
Intelligence is worthless without an intelligent decision-making process. During the run-up to the second Iraq War, the CIA sent Iraqi-Americans related to Iraqi nuclear scientists to inquire about the status of that country's nuclear program. Thirty Iraqi-Americans were debriefed by the CIA and sent independently of each other. All thirty returned with news that the Iraqi nuclear program had been run into the ground by a relative of Saddam, that the scientists lied about their progress to Saddam to stay in his good graces, and that Saddam was bluffing by denying UN inspectors.
In fact, a few scientists reported that Iraq had no real capability to make nuclear bombs since the early nineties. A crucial centrifuge facility had been destroyed in the first Gulf War. The facility had been unknown to Western intelligence until Saddam's hand-picked boss ordered it to be moved to a safer location. American intelligence detected the activity. They didn't realize that was a nuclear processing facility but knew it was a military target. Thus, the facility was put as a secondary target on the Air Tasking Order designating targets for air bombardment.
One day, a fighter-bomber returning to its carrier had unexpended laser-guided munitions because its primary target had been masked by weather. Back then, American planes could not land with unexpended munitions because the explosives were not inert and posed a risk of fire or explosion. The air traffic controller directed him to the nuclear facility. The bombs hit their target and that was the end of the Iraqi nuclear program.
Hate to Godwin here, but this is old problem (Score:1, Interesting)
Indeed. In addition, this is not a new problem.
If you visit STASI museum* in Berlin, one thing you'll no doubt learn is this: STASI was a lot more inefficient than they could have been because they had much more data than they could process. They didn't have problem with gathering information but with acting based on it. They received so many reports and pieces of information that there was no chance for them to tie them all together and analyze them all.
That's why I don't think that the biggest risks we face are increase in amount of CCTV cams and whatnot. They can't effectively be used to trace us because nobody can analyze all that data. Solving crimes (as inefficient as they are in that) is pretty much the only thing they can be used for. However, when I hear about new advances in areas of face recognization and stuff like that I always begin wonder what kind of central intelligence agency will the next dictatorship have access to.
*Yes, I know the difference between Gestapo and STASI but if you interpret godwin's law that literally, you are missing the point. In addition, I would be surprised if Gestapo didn't have a similar problem.
Re:America forced Japan's hand (Score:3, Interesting)
I've got to disagree with you almost completely on this one. Japan made a slick tactical maneuver with their attack on Pearl Harbor but a huge strategical blunder. If Japanese war planners had paid any attention to their own intelligence, they would have realized that even if they had succeeded in knocking out the American Pacific fleet, all that would have accomplished was delaying the inevitable.
As it was, all the Pearl Harbor attack accomplished was to give the Imperial Japanese Navy six months of free reign in the western Pacific. It wasn't enough. The industrial capacity of the U.S. dwarfed that of Japan, which the Japanese military had known but chosen to ignore. The reasons given for that vary--perhaps arrogance, possibly belief in a divine destiny but no matter how you spin it, it was a strategic mistake.
And, by the way, the U.S. was not actively involved in the war in Europe at the time of the Pearl Harbor attack, although it was supplying weapons and materials to Great Britain to help them in their fight against Germany.
Re:America forced Japan's hand (Score:2, Interesting)
If the Japanese had succeeded in wiping out the Pacific fleet in one fell swoop, it would have unlocked the entire Asian continent for them. American was already fighting a war on the European front. A crippled Pacific fleet would have made it impossible to keep up a second front.
No, it wouldn't. The bottom line is that our industrial capacity was just too much for them. It wouldn't matter what they did in the opening stages; we would have out-produced them and defeated them eventually no matter what. And there were more than enough production resources to smother both the Germans and the Japanese. (I know, technically the Soviets did most of the smothering of the Germans, but that's beside the point here).
The fundamental problem with the Axis powers is that they just weren't big enough to take on the whole rest of the world, and that their nationalistic/racist ideologies made it impossible to build true alliances. The only thing that might have actually worked for both Germany and Japan would have been to conquer just enough territory that nobody would complain too much, then consolidate and industrialize that territory, and repeat until they have a bigger war machine than their rivals. But that probably isn't possible if you believe that the citizens in all of the neighboring countries are all sub-human.
Re:HUMINT SIGINT (Score:3, Interesting)
An individual person can't, the volume is too large, and the information is often dispersed. However, large networks of people could potentially gather and bring together that intelligence in a timely fashion so that it can be acted on. Obviously, this didn't happen before Pearl Harbor, or 9/11. But can this type of intelligence gathering be done in the real world- can we collect dispersed information, bring that information together, and do in hours, rather than days? The answer is "yes", and to see why, just look at the DARPA balloon challenge.
I really didn't get what DARPA's red balloons were all about until I read this article. It seemed sort of abstract, something about social networks. I suspect that this question (how do we collect and assemble those needles of intelligence from a vast haystack of noise?) is the question that DARPA is trying to answer with those balloons. In the case of Pearl Harbor, or 9/11, or the Fort Hood shootings, there was actionable intelligence. The problem was that there wasn't a mechanism to collect that intelligence; the people who knew the facts, and the people who needed the facts, didn't know each other. What DARPA asked was: how can we collect intelligence when the intelligence is held by different groups of people (think different government agencies, like CIA, NSA, FBI, Customs and Immigration, different governments, or just people in the streets) and those people don't know the people who need the information (higher-ups in Homeland Security or the White House)? We know what failure looks like (Pearl Harbor, 9/11) but what does success look like? How and why do certain intelligence-gathering systems actually work, when they do work?
Their unconventional approach here was to set up an intelligence problem (balloons dispersed all over the country, need to collect info within 24 hours) and then let other people figure out how to solve it. Obviously terrorists will not be painted bright red, clearly marked, and stationary, but the principles of effective network intelligence can be applied to more difficult problems. I suspect that DARPA is going to spend a lot of time studying the data about how the intel came in for these various efforts.