In 2004 Jeff Hawkins' book On Intelligence was widely praised. But Hawkins made some claims about artificial neural networks that seem (to me) untenable today, only 13 years later. He gives the impression that artificial neural networks are in principle not capable of what he calls invariant representations and hierarchical storage of patterns (p. 70 ff).
My questions are:
Did he just not (fore)see the development of deep learning which started off at the latest in 2005 (see History of deep learning) and which includes both invariant and hierarchical representations?
Or are his retentions against artifical neural networks (including deep learning) justified, and the invariant and hierarchical representations of deep learning are uncompetitive to those of biological networks, i.e. the neocortex? In which respect, then?
About invariant and hierarchical representations (in my words and understanding):
An invariant representation - e.g. of a face - is one, that is independent of and thus invariant with respect to contingent aspects of the image of the face having to do with distance, viewing angle, lighting conditions and the like.
It is the representation in the highest layer of a hierarchy of representations with local geometric features being represented in the lowest layer.
This describes superficially both deep learning and cortical architectures. But maybe there are details of implementation/realisation that make the one significantly and genuinely more efficient than the other, e.g. cortical columns which seem not to be present in deep learning architectures, do they?