Can an Intelligence Test for Machines Be Designed?

Intelligence tests for humans, intelligence tests for machines. How different can they be?

Well, yes, the machines don't need bathroom breaks or #2 pencils, but aside from that, the disputes that have roiled the world of standardized testing would not be unfamiliar to your average robot, were she/it properly programmed to begin with.

The questions have not just to do with how the measures are being designed and implemented but for what purpose they are intended.

As anyone who has sat in on an intellectual discussion can imagine, there is not only significant disagreement about this, but a raft of less-than-generous commentary about the efforts of others who have attempted the feat.

This work will eventually be perfected, which is to say, an approach satisfactory to enough people to give it a chance of being improved iteratively will be shared. At which point, even more arguments will break out, but competitive juices will drive development from that point. It may not be the most intelligent path, but then that depends on how you define intelligence. JL

Jacob Aron reports in New Scientist:

Whatever the best approach, it's clear that tech companies like Facebook and Microsoft are betting big on human-level AI. Should we be worried?

John is in the playground. Bob is in the office. Where is John? If you know the answer, you're either a human, or software taking its first steps towards full artificial intelligence. Researchers at Facebook's AI lab in New York say an exam of simple questions like this could help in designing machines that think like people.

Test yourself on some of the questions here.

Computing pioneer Alan Turing famously set his own test for AI, in which a human tries to sort other humans from machines by conversing with both. However, this approach has a downside.

"The Turing test requires us to teach the machine skills that are not actually useful for us," says Matthew Richardson, an AI researcher at Microsoft. For example, to pass the test an AI must learn to lie about its true nature and pretend not to know facts a human wouldn't.These skills are no use to Facebook, which is looking for more sophisticated ways to filter your news feed. "People have a limited amount of time to spend on Facebook, so we have to curate that somehow," says Yann LeCun, Facebook's director of AI research. "For that you need to understand content and you need to understand people."

AI plays 20 questions

In the longer term, Facebook also wants to create a digital assistant that can handle a real dialogue with humans, unlike the scripted conversations possible with the likes of Apple's Siri.

Similar goals are driving AI researchers everywhere to develop more comprehensive exams to challenge their machines. Facebook itself has created 20 tasks, which get progressively harder – the example at the top of this article is of the easiest type. The team says any potential AI must pass all of them if it is ever to develop true intelligence.

Each task involves short descriptions followed by some questions, a bit like a reading comprehension quiz. Harder examples include figuring out whether one object could fit inside another, or why a person might act a certain way. "We wanted tasks that any human who can read can answer," says Facebook's Jason Weston, who led the research.

Having a range of questions challenges the AI in different ways, meaning systems that have a single strength fall short.

The Facebook team used its exam to test a number of learning algorithms, and found that none managed full marks. The best performance was by a variant of a neural network with access to an external memory, an approach that Google's AI subsidiary DeepMind is also investigating. But even this fell down on tasks like counting objects in a question or spatial reasoning.

Richardson has also developed a test of AI reading comprehension, called MCTest. But the questions in MCTest are written by hand, whereas Facebook's are automatically generated.

The details for Facebook's tasks are plucked from a simulation of a simple world, a little like an old-school text adventure, where characters move around and pick up objects. Weston says this is key to keeping questions fresh for repeated testing and learning.

But such testing has its problems, says Peter Clark of the Allen Institute for Artificial Intelligence in Seattle, because the AI doesn't need to understand what real-world objects the words relate to. "You can substitute a dummy word like 'foobar' for 'cake' and still be able to answer the question," he says. His own approach, Aristo, attempts to quiz AI with questions taken from school science ex Recently the likes of Stephen Hawking, Elon Musk and even Bill Gates have warned that AI researchers must tread carefully.

LeCun acknowledges people's fears, but says that the research is still at an early stage, and is conducted in the open. "All machines are still very dumb and we are still very much in control," he says. "It's not like some company is going to come out with the solution to AI all of a sudden and we're going to have super-intelligent machines running around the internet."