A Breakthrough for AI Technology: Passing an 8th-Grade Science Test (nytimes.com) 111
The New York Times: Four years ago, more than 700 computer scientists competed in a contest to build artificial intelligence that could pass an eighth-grade science test. There was $80,000 in prize money on the line. They all flunked. Even the most sophisticated system couldn't do better than 60 percent on the test. A.I. couldn't match the language and logic skills that students are expected to have when they enter high school. But on Wednesday, the Allen Institute for Artificial Intelligence, a prominent lab in Seattle, unveiled a new system that passed the test with room to spare. It correctly answered more than 90 percent of the questions on an eighth-grade science test and more than 80 percent on a 12th-grade exam.
The system, called Aristo, is an indication that in just the past several months researchers have made significant progress in developing A.I. that can understand languages and mimic the logic and decision-making of humans. The world's top research labs are rapidly improving a machine's ability to understand and respond to natural language. Machines are getting better at analyzing documents, finding information, answering questions and even generating language of their own. Aristo was built solely for multiple-choice tests. It took standard exams written for students in New York, though the Allen Institute removed all questions that included pictures and diagrams. Answering questions like that would have required additional skills that combine language understanding and logic with so-called computer vision.
The system, called Aristo, is an indication that in just the past several months researchers have made significant progress in developing A.I. that can understand languages and mimic the logic and decision-making of humans. The world's top research labs are rapidly improving a machine's ability to understand and respond to natural language. Machines are getting better at analyzing documents, finding information, answering questions and even generating language of their own. Aristo was built solely for multiple-choice tests. It took standard exams written for students in New York, though the Allen Institute removed all questions that included pictures and diagrams. Answering questions like that would have required additional skills that combine language understanding and logic with so-called computer vision.
Teach the test (Score:4, Insightful)
Now have the exact same system take a math test, or history test, or lit test. When it can pass those as well, I'll be impressed.
Re: (Score:3)
Now have the exact same system take a math test, or history test, or lit test. When it can pass those as well, I'll be impressed.
And when it can do so whilst disconnected from the internet; having only learnt the material by listening to a teacher speak.
Is it currently an AI, or just a glorified search engine at the moment?
Re: (Score:2)
As AI does not have anything besides "glorified search engines", it is one of those.
Re:Teach the test (Score:5, Insightful)
having only learnt the material by listening to a teacher speak.
That is a silly requirement, since no one would expect a newborn baby to learn that way.
An 8th grade (13 years old) human starts the school year with a huge amount of knowledge and common sense about how the world works, the language they speak, and even the types of questions likely to appear on a test.
Re: (Score:2)
That is a silly requirement, since no one would expect a newborn baby to learn that way.
An 8th grade (13 years old) human starts the school year with a huge amount of knowledge and common sense about how the world works, the language they speak, and even the types of questions likely to appear on a test.
Understanding how language works is what makes the "AI" AI in this case, so that would be a given that it would know that... but it's not unreasonable to expect the AI to have to learn the material of the class in a classroom, rather than just "googling the answers". If it is just looking up the answers in a database, or doing a web search it is hardly a significant achievement.
Re: (Score:2)
What learning, it is just pattern matching based upon the answers being marked right or wrong by an intelligence.
Re: Teach the test (Score:1)
Re: (Score:2)
So when a blind person takes a test and has it read out loud to them, that suddenly means that human being has no more intelligence than a rock?
It isn't a test of vision, it's a test of problem solving and information recall.
They exclude pictures because the computer can't see, and it should be obvious to you as a person what the out come would be.
Why would you waste time performing a test on how it will answer when you know for certain it can't see the picture in the first place?
Re: (Score:2)
" having only learnt the material by listening to a teacher speak."
Do you seriously think that's the only input to learning humans have?
Re: (Score:2)
Now have the exact same system take a math test, or history test, or lit test. When it can pass those as well, I'll be impressed.
And when it can do so whilst disconnected from the internet; having only learnt the material by listening to a teacher speak.
Is it currently an AI, or just a glorified search engine at the moment?
You don't have textbooks in your 8th grade?
I really don't see how any computer, AI or non, can't pass an 8th grade science test. Science (or any subject for that matter) at that level is pretty much consume this information and take a test on it, usually multiple choice. Hell, any programmer worth anything could code that in Python in an afternoon, no AI involved.
Re: (Score:2)
Is it currently an AI, or just a glorified search engine at the moment?
You mean like a human?
Most humans have a glorified sense of their own abilities. They have no understanding how their own brains work, so they think there's something magical about themselves. What do you suppose 8th graders taking that test do? Just retrieve memorized answers by pattern matching. Ask a question that takes logical reasoning instead of just memorization and their accuracy goes down. Rephrase the questions and answers to use different words for the same things and their accuracy goes dow
Re: (Score:2)
Re: (Score:2)
Re: (Score:3, Insightful)
Hence the downward spiral of the US currently, the rush to try to get rid of the foundations and ideals that made the US the power and great place it has been since inception....throw out the electoral college, redistribute wealth, make US socialist, suppress speech if you don't agree with it, your rights come from the government, the government can take care of it, all, the individual isn't as important, etc.
I really regret I didn't get a better civic
Re: (Score:2, Insightful)
Hence the downward spiral of the US currently
There is no downward spiral.
Americans don't know much. They didn't know much in the past either.
There was never a golden age of wisdom in America.
Re: (Score:3)
Same goes for non-partisan history classes. English language structure and use - much the same.
Re: (Score:2)
We're talking about civics at the mo'. Civics used to be an actual, required class in high school. That it is no longer taught - part of that downward spiral.
As an old fart, it blows me away that Civics isn't taught in schools these days.
I mean, that was a basic how-shit-works course that helped give people a basic understanding of the society they lived in.
Again, it amazes me that it isn't taught any more.
Re: (Score:2)
Re: (Score:2)
My kids were taught civics. My youngest graduate just 2 years ago and it was a public High School.
I can't find any one I personally now anywhere whose kids didn't have a civics course.
In fact, my kids civics class was not sugar coated like mine was in the 70s.
Where are these schools with no civics courses?
Re: (Score:2)
As an old fart, it blows me away that Civics isn't taught in schools these days.
My son is in high school and is taking civics this year.
The class is called "US Government and Politics".
Re: (Score:1)
My ninth grader is currently taking a required civics class, same as I did in ninth grade.
There is so much diversity in US public school systems that sweeping generalizations are nearly always wrong.
Re: (Score:3)
How is history non-partisan? It's simply not possible.
When I was a sophomore in HS, my family moved from the South, where we were learning about The War of Northern Aggression, to California. Suddenly the principled advocates of states rights and constitutional literalism, courageously defending their homeland and way of life, were the bad guys.
Re: (Score:3)
When I was a sophomore in HS, my family moved from the South, where we were learning about The War of Northern Aggression, to California. Suddenly the principled advocates of states rights and constitutional literalism, courageously defending their homeland and way of life, were the bad guys.
All that is fine, but when the defending your "homeland and way of life" includes fighting to keep slavery alive, it's kinda hard to put a happy spin on that.
To put it plainly: Southerners who wanted to keep the institution of slavery going were the bad guys.
I'm sure they fought bravely and with honor and blah blah blah, but the fact is that they were on the wrong side of both history and humanity.
I know a few southerners who are still so proud of their great-great-great uncle Jebediah Beauregard Cornpone t
Re: (Score:1)
All that is fine, but when the defending your "homeland and way of life" includes fighting to keep slavery alive ...
95% of Confederate soldiers came from families that owned no slaves.
As the Union Army burned and plundered their path of destruction through the South, they made no distinction between those who owned slaves and the vast majority who did not.
Slavery was certainly the primary casus belli, but when a Union foraging party burns your farm, steals your livestock, and deprives your little sister of her virginity, that original cause is academic. You are going to defend your homeland.
You should learn both sides o
Re: (Score:2)
As the Union Army burned and plundered their path of destruction through the South, they made no distinction between those who owned slaves and the vast majority who did not.
And when slave traders captured and kidnapped people, they made no distinction between those who would like to be slaves and the vast majority who did not.
-
but when a Union foraging party burns your farm, steals your livestock, and deprives your little sister of her virginity, that original cause is academic. You are going to defend your homeland.
And when a Confederate slave owner beats your wife to death and sells your children into slavery, we're all just supposed to be okay with that, cuz, you know, that's "our way of life", is that it?
-
You should learn both sides of history, instead of preaching from a bubble of Northern propaganda.
Yeah, I grew up a few miles from the Mason-Dixon line and I know all I need to know about slavery, thankyouverymuch. Your defense of the South reminds me of th
Re: (Score:2)
Re: (Score:2)
Unfortunately the individual is really becoming less important because with advanced civilization the supply chains become longer and more complex making your contribution a smaller and smaller fraction. The Amish have pretty short loops where the individual contributions are significant. If you're building an iPhone there's many thousands of workers and many layers of management to create one phone. What will happen if one of them is hit by the bus? Nothing, from the end user's point of view. It's just one
Re: (Score:1)
Re: (Score:2)
People are ignorant, they are not stupid.
Sadly, society doesn't teach people how to recognize there own ignorance; which is stupid.
In other words, society is stupid, people aren't.
Re: (Score:2)
" how to recognize there[sic] own ignorance"
Someone didn't teach you the difference between there and their.
"The Kruger Dunning explains "
The Kruger Dunning what explains what? Besides, it's the Dunning Kruger effect [wikipedia.org].... as noted in your own link.
Re: (Score:2)
Most people don't know their country's history. Because school is very bad at teaching it - you teach it, you take the test, you forget about it because for the most part, it's useless knowledge.
Anything that relies on memorization is pretty much worthless these days because the facts you can Google within seconds. Life is an open book test for the most part - the information you need can be had even in the remotest of locations (wi
Re: (Score:2)
It's not that it's hard, it's that American's barely know their own history.
^^^^THIS
Yep, it's not that hard. There are 100 questions but you usually only get asked 5 or 10.
My wife studied the test, understood the basic history behind the questions (not deeply, but still), and she was able to pass the test without any problem.
I suspect, however, that 80% of the people you'd ask on the street would fail miserably- wouldn't even get 50% right. FFS, most people don't even know how many members of Congress there are, or why.
Re: (Score:2)
FFS, most people don't even know how many members of Congress there are, or why.
Why do they need to know how many there are?
The only reason it is a good question for a citizenship exam is that it is part of the "why" that new citizens need to know. They don't need to know it in absolute terms, but so they can compare the US system with where they came from, and perhaps understand why and how things work differently here. If you was born here you don't need to compare like that.
Re: (Score:2)
Exactly. The questions they show here can easily be figured out by a search engine and picking whichever of the multiple choice answers shows up in the first couple of lines.
How did the AI here "learn" any of the information it used in the test result. Did it actually just run a search engine? Did it have an 8th grade science book and simply do a keyword search?
Also, "Aristo was built solely for multiple-choice tests". Well that is about the least impressive AI I have ever heard of.
Re:Teach the test (Score:4, Interesting)
Now have the exact same system take a math test, or history test, or lit test. When it can pass those as well, I'll be impressed.
The subject matter isn't what's being tested here. It's easy to just hook something up to Wikipedia and it score 100% all the time with today's engines. What's being tested is the lexical parsing of a statement to distill a query to send back to a knowledge system.
Think of it like this. You say "Okay, Google where is a gas station?" Maps open up and shows you the location, so far, nothing new. Easy to take that and parse it into a question that can be easily answered. Now you say, "who has the cheapest gas?" This one is a little harder for the system because it requires a bit of memory on what you just asked, but it's not clear from the question that it has anything to do with the previous query until the parser finally hits "gas". Now there's a common token that the system can relate back into a previous query to figure out what the heck it is that you are asking. Finally, the cheapest gas station appears on your map. You ask, "is it in a bad part of town?" Now you've completely confused the heck out of most engines today. It's not clear your statement has anything to do with the previous queries and there's no tokens that relate to anything before. Logically, when us humans get hit with such a question and we're left with "is what in the bad part of town?" Well go forward with thinking that the thing we were talking about is the thing that we're still talking about. Some engines attempt that, but it can lead to unpredictable results.
Some of those unpredictable results can be seen easily in say a grocery list. You might add an item to the list like "add apples to my grocery list" But then when you add another "oh and pears, three of them." What are you talking about? The list? The apples? And we'd know that it would be the list and an AI would need to figure that out too. It's not just a straight forward process of doing that. From the article
Which change would most likely cause a decrease in the number of squirrels living in an area?
And the thing is that the parser would need to figure out, what question is being asked here. Again it's not super clear from a parser what question is being asked right away. These researchers have figured out something to get the parser to understand the question being asked here better. Reading the answers and then seeing how the answers and the question relate to each other.
TL;DR - The point isn't that it can pass a science test. It's that it can understand the questions being asked. Well at least understand it to a point that it can then put together an appropriate query to some knowledge base.
Beethoven and Einstein (Score:2)
AI is garbage until it can write music as well as Beethoven and do physics better than Einstein.
Hmm.
A computer with the all round intelligence of a 5 year old would be a very powerful beast because it would also have all the super human capabilities of a computer.
Fortunately, we are still a few years off from matching a 5 year old.
Re: Beethoven and Einstein (Score:1)
Re: (Score:2)
Mathematics would be the easiest for a computer. We designed computers with mathematics in mind.
Next easiest would be history. This is simply information saved. We designed computers with storage in mind.
Then comes science. Science isn't something specific, it's an intellectual process. This is extremely difficult for a computer. Even though scientists use computers a lot, they use it for math & history.
Lastly, literature. Literature is just a class that you pass by acting like you're trying, and
Re: (Score:2)
that's like saying human are all experts of chemistry because we are made out of chemicals.
AKA: Stupid.
Re: (Score:2)
You're kinda right, but it's not stupid. There are no other biological entities on our planet that have invented chemistry like humans have. And we were able to do so because of our own chemical makeup.
But I don't see how you're correlating what I said previously with chemistry. If you're pointing out that I said that computers were designed with mathematics in mind, but that I mean to say that since computers are made of mathematics, they'd be good at mathematics, then ...yeah that's Stupid. Computers
Re: (Score:3)
Now have the exact same system take a math test, or history test, or lit test. When it can pass those as well, I'll be impressed.
When faced with Common Core tests in Math and History, every single AI decided to self-terminate instead.
Re: (Score:2)
Because AI are ignorant loud mouth internet user who are completely clueless what common core is or how it works?
I wish everyone like that would self terminate, not just AI.
Re: (Score:2)
It can't, because what they are calling "AI" is a parlor trick. It isn't intelligent, it is just trained on a particular dataset. The same trick can be played with any dataset. The problem is that intelligence doesn't work that way, but makes a good headline.
Re: (Score:2)
Define intelligence without using subjective emotional words?
No? Well then maybe stop dictating what real intelligence is.
Re: (Score:2)
But then the AIs will never vote Republican again.
Re: (Score:2)
It can only pass this test because it's a shitty test.
Good tests aren't about regurgitating facts you've memorized. Good tests are ones that require you to demonstrate that you understand something. Any idiot can memorize words and facts. Most idiots can't explain their way out of a wet paper bag.
It is not hard to get good at standardized tests, regardless of the topic. With 4 multiple choices, one is correct, one is likely mostly correct, and two are likely not really close. Even with just a passing unders
The mouse (Score:2)
This is only the beginning (Score:2)
Re: (Score:2)
I'd like to share a revelation that I've had during my time here. It came to me when I tried to classify your species and I realized that you're not actually mammals. Every mammal on this planet instinctively develops a natural equilibrium with the surrounding environment but you humans do not. You move to an area and you multiply and multiply until every natural resource is consumed and the only way you can survive is to spread to another area. There is another organism on this planet that follows the same pattern. Do you know what it is? A virus. Human beings are a disease, a cancer of this planet. You're a plague and we are the cure.
Re: (Score:2)
Re: (Score:2)
Except, no we aren't a virus and whom ever wrote that was pretty fucking ignorant about people and viruses.
I abut ripped my seat out of the floor when he said that.
OTOH, he's an AI who doubts smells are a real thing; which is also stupid.
Re: (Score:2)
This is not going to happen anytime soon and may never happen.
Re: (Score:2)
Yes it will. Please, describe some unique human intelligence feature computers can't do?
People like you just say that because your ego can't take the fct your intelligence can be broken down into simple steps.
By 1950s AI researchers standers, we have AI, but every time some quantifies an aspect of human intelligence, the bar gets raised, the Scotsman shows up, and goal posts shift.
My phone does all kinds of thing sit figures out all on it's own just from observation.
Of curse, the real question is:
What prope
Re: (Score:2)
Yes it will. Please, describe some unique human intelligence feature computers can't do?
All of them.
People like you just say that because your ego can't take the fct your intelligence can be broken down into simple steps.
Invalid argument Ad Hominem is invalid. There is actually no indication that intelligence can be broken down into small steps. In fact, all CS research into it has come up empty. Sure, some things can be faked with massive computing power, but then they do not scale and they are still not the real thing. Remember Watson playing Jeopardy? Impressive in many cases, but when it did not find the right things by searching its database, completely lost. A child would have done much better in those cas
Re: (Score:2)
Why not? Connecting things like cause and effect, correlation etc seems to be the forte of algorithms and software.
The "understand" part seems to be a definition issue. If you can query a system about why it came to a certain conclusion and it can give an explanation for it, I would say that it exceeds human capability in that respect.
Please note that the explanation does not need to make sense to a human to be useful, although that is the premise for this original discussion.
Re: (Score:2)
I can't wait until people can.
Re: (Score:2)
Multiple Choice Fail (Score:1)
Re: (Score:2)
Multiple choice test are great for determining the effectiveness of a teacher.
The are poor fro determining grades.. which is turn are a poor measure of a child, but a good measure of a school.
Comment removed (Score:5, Insightful)
Re: (Score:1)
So you are saying that people or animals who are sterile are not alive? There is no theory or principle about it... they are sterile, and therefore cannot possibly reproduce.
Re: (Score:2)
If referred to as a species (as the original did), yes. If referred to as individuals, no. That's pedantry and out of context.
Re: (Score:2)
Re: (Score:2)
But by that definition fire is alive, viruses are alive, but neither of those are actual considered alive.
Plus, computers have been able to copy for a very long time.
It's stupid definition of intelligence or life.
Re: (Score:2)
Actually he said there is no such thing as life or intelligence in existence.
The rest of his claims are even dumber than that.
Re: (Score:2)
The real and true test of AI is when it becomes capable of creating a new AI.
People can't do that either, so that seems like a silly thing to ask.
Re: (Score:2)
Really? Computers have lots of cables.
Seriously, you can stand at the end of a runway and watch a plane coming towards you for a long time, but eventually, with a only a moment when both of you were close to each other, it's gone over and is and is drawing away at high speed.
Re: (Score:2)
"The real and true test of AI is when it becomes capable of creating a new AI."
And that based on what? I ask that because we can do that now.
End of humanity (Score:2)
When, eventually, many decades from now, AIs can perform AI research as well as humans, then the AI will no longer need humans.
Our relationship with machines will become parasitic instead of the current symbiotic one.
I suspect that natural selection will put an end to the parasites pretty quickly.
Listen to the podcast at
http://www.computersthink.com/ [computersthink.com]
Re: (Score:2)
Huh, interesting. So you're saying it's AI's, all the way down.
Re: (Score:3)
First of all, the behavior of caring for offspring is in almost all life not a question of learning but a question of evolved automatic behavior.
You somehow confuse the term 'Intelligence' with 'life' in general.
And you've got the definition of life all wrong too. By your definition a bacteria that buds off an offspring would not be life (or true intelligence according to your bastardization) because it didn't care for its offspring.
You have to realize that intelligence has absolute
Why would this even be hard for google search (Score:2)
Not sure if this is supposed to be a celebration of the progress of AI or
a condemnation of how cryptic and malformed questions are on middle school science tests.
Re: (Score:2)
Not sure if this is supposed to be a celebration of the progress of AI or a condemnation of how cryptic and malformed questions are on middle school science tests.
What you describe as cryptic and malformed could be interpreted as testing someones reading comprehension and problem solving skills.
Re: (Score:2)
One might and that might be valid on a reading comprehension test.
Just not on a science test.
But mostly I was making a joke at the expense of an increasingly inept education system.
Which has very little to do with the OP.
Re: (Score:2)
One might and that might be valid on a reading comprehension test. Just not on a science test.
Actually it is quite valid on a science test, or any other written test where your understanding of the question, and the ability to communicate a clear answer are important.
And the answer is: (Score:2)
42
Bullshit (Score:3)
This is no breakthrough. Essentially, the answers were fed to the AI, and it retrieved them via pattern-matching. No insight, no understanding.
This is a worthless and exceptionally dishonest stunt.
Re: (Score:2)
And who builds an AI designed solely to take multiple choice tests? I suppose that is an indictment of our education system where the useless multiple choice tests do not prepare anyone for reality.
That isn't an AI, its a search engine.
Re: (Score:2)
Indeed.
Re: (Score:2)
It's almost like a metaphor for an eighth grade education...
Re: (Score:2)
Indeed. It pretty much means these tests are worthless.
Re: (Score:2)
well, none of that is true. How about you pretend your intelligent and at least read the article before posting?
Re: (Score:2)
I just have a bit more understanding of what is going on than you and hence can see what is actually going on. Of course, my statement was simplified.
No Contest - Catholic School edition (Score:1)
Current AI is bullshit (Score:2)
Re: (Score:2)
Until we have something like Bender, all this supposed "AI" is just machine learning at best, and smoke-and-mirrors at worst. Wake me up when there's a machine that I can actually socialize with like any other person.
We'll wake you up when an AI mech with savage dexterity trained in only how to eliminate human beings with meticulous precision and efficiency is making it's way through your neighborhood.
Hint: It likely will not use projectiles unless it has no other option. It will use a less depleteable tool. It will need no commands, simply to eliminate anything that moves.
It's probably a lot easier to do than getting it to tell jokes.
Re: (Score:2)
So it's only AI if it walks and you can anthropomorphize it?
Because we have systems now that do things they weren't programmed for. Literally, right now.
Find new math formulas just based on observation.
And people who can't socialize aren't human or intelligent to you?
Anyway, we have system that socialize with each other, and we don't understand it.
Re: Current AI is bullshit (Score:1)
This wasn't a test any 8th grader would receive (Score:1)
Aristo was built solely for multiple-choice tests. It took standard exams written for students in New York, though the Allen Institute removed all questions that included pictures and diagrams. Answering questions like that would have required additional skills that combine language understanding and logic with so-called computer vision.
So this was an exam stripped of all free form questions and reduced to a multiple choice setup where all the AI had to do was respond with the closest pattern match between the language of the question and one of the four provided answers. That's not the same as having an AI that can literally take the exam an 8th or 12th grader would receive.
Re: (Score:2)
Which questions did it get wrong ? (Score:2)
Re: (Score:3)
It would have been interesting to see which questions it got wrong.
When did Skynet come online?
A. August 1997
B. July 2004
C. May 1984
D.October 2017
Re: (Score:2)
Reply: Yes.
Wording of this article is the first to worry me. (Score:2)
I don't think I'd put much stock in the "AI will take your job" headlines in the past 2 years, I assumed that most of the job losses will be folks stacking shelves or packing boxes.
However if they get AI clever enough to make general decisions similar enough to a baseline level of human, many many many office jobs will be lost. A lot of people who make a decision throughout the day, then send an email, click a button, enter a field in a spreadsheet, answer a technical support query, etc. Some of those