Question concerning Jeff Hawkins' On Intelligence

Question

In 2004 Jeff Hawkins' book On Intelligence was widely praised. But Hawkins made some claims about artificial neural networks that seem (to me) untenable today, only 13 years later. He gives the impression that artificial neural networks are in principle not capable of what he calls invariant representations and hierarchical storage of patterns (p. 70 ff).

My questions are:

• Did he just not (fore)see the development of deep learning which started off at the latest in 2005 (see History of deep learning) and which includes both invariant and hierarchical representations?

• Or are his retentions against artifical neural networks (including deep learning) justified, and the invariant and hierarchical representations of deep learning are uncompetitive to those of biological networks, i.e. the neocortex? In which respect, then?

---

About invariant and hierarchical representations (in my words and understanding):

• An invariant representation - e.g. of a face - is one, that is independent of and thus invariant with respect to contingent aspects of the image of the face having to do with distance, viewing angle, lighting conditions and the like.

• It is the representation in the highest layer of a hierarchy of representations with local geometric features being represented in the lowest layer.

This describes superficially both deep learning and cortical architectures. But maybe there are details of implementation/realisation that make the one significantly and genuinely more efficient than the other, e.g. cortical columns which seem not to be present in deep learning architectures, do they?

Answer 1

Summary

I think Hawkins' terminology tends to be not as precise as it should be. I guess that's the root cause of this question as well.

The impression that Hawkins says computers or neural networks will never be able to perform certain tasks which are routine for the human brain may be due to his lack of precision.

Details

You write (in a comment):

I have the impression, that he sees computers and ANNs as somehow the same.

That's not my impression¹, so I'll answer two questions:

1. Regarding Hawkins' statements about artificial neural networks
2. Regarding Hawkins' statements about "computers"

Regarding Hawkins' statements about artificial neural networks

In chapter 2, "Neural Networks", pp. 23-39, Hawkins talks about a certain kind of artificial neural networks (mainly pp. 24-29).

He's mostly talking about very simple ANNs. See p. 25:

Most neural networks consisted of a small number of neurons connected in three rows.

It is true that there are a lot of tasks that such simple ANNs can't do (or can't do well), but of course that's not true for more complex ANNs. I think the problem here is that Hawkins' doesn't always make this distinction clear enough, although he says on p. 28:

I don't want to leave you with the impression that all neural networks are of the simple three-layer variety. Some researchers have continued to study neural networks of different designs. Today the term neural network is used to describe a diverse set of models, some of which are more biologically accurate and some of which are not.

Maybe the following quote on p. 32 could be interpreted such that he actually did foresee some developments after 2004:

In recent years, belief in the importance of feedback, time, and prediction has been on the rise. But the thunder of AI and classical neural networks kept other approaches subdued and underappreciated for many years.

Regarding Hawkins' statements about "computers"

In chapter 4, "Memory", pp. 65-84, (which contains p. 70ff you mentioned) he does not talk about artificial neural networks, but about "computers". He doesn't specify exactly what he means, but I guess it's roughly "machines that work mostly like current computers".

On p. 67, he says that even "the largest parallel computer imaginable can't solve" certain tasks that the brain can solve. I think that's wrong, but since it's not quite clear what he means by "computer", I'm not sure.

Conclusion

Some or most of Hawkins' claims may be correct, but that depends on what he means by "computers" or "neural networks", and he doesn't always make that clear.

Footnotes

¹ For example, this quote on p. 24 seems to imply that he does see a difference:

A neural network is unlike a computer in that it has no CPU and doesn't store information in a centralized memory.