How to compare tasks completed by neural architectures objectively?

Question

When I first saw this video of Spaun and the tasks it can complete (solving the Towers of Hanoi problem, completing the Raven matrices), I was really impressed, but then I realized I didn't really know what other tasks were being completed by other neural architectures. Consequently, I almost asked a question here that was basically "How does Spaun compare in terms of task complexity to other neural architectures?". However, I realized that it would be better to if I could learn how to do this evaluation myself.

So, what metrics or heuristics can be used to evaluate the complexity of a task completed by a neural architecture? How can one evaluate the complexity of a problem/puzzle such at the Tower of Hanoi, to another problem, such as modelling how people solve algebraic equations, like ACT-R has accomplished?

Answer 1

More than two years later, I now have a heuristic (which is a close as I can get to an objective metric) that I rely on. As I write about in one of my blog posts, there are a couple of tasks that humans are capable of that are still really hard for machines. They require the following things for a machine (or a neural model):

• Balancing Planning and Exploration: Modelling the environment's future state and the possible future states via exploration efficiently.
• Scaling Skills: Transfering skills between tasks and building more complex skills atop of basic skills.
• Knowledge Representation: Keeping relations between skills, environmental variables, past knowledge and priorities all usefully organised.

Any neural model that begins to leverage these things is evaluated as impressive in my eyes.

However, these requirements are mostly for Artificial Intelligence. What about neural models specifically? All neural models that complete a task are, by definition, cognitive models, so in addition to the above criteria they must satisfy various Cognitive Criteria to prove the system is achieving it's goals in a human-like manner.