# How are positions and counts of higher concepts encoded in sparse representations?

I understand that levels in the sensorimotor hierarchy of the brain learn recurring features of their input streams. The results are activation patterns with single neurons firing for the existence of specific higher level patterns. Those sparse representations are passed upward to the next higher layer.

For example, there is a neuron firing whenever there is a face in the scene. On this higher level, neurons are not restricted to a small window of our field of view anymore. My question is how those high level sparse representations still retain information about positions and counts of objects.

One bad explanation would be that there are neurons representing counts in general. If "three" and "face" are both active, there are three faces in the scene. But this doesn't hold for two kinds of objects at the same time since it's not clear which count is for which object. And what if both have the same count.

• Your title seems to be missing some words: "How are positions and counts of higher _____ _____ encoded in sparse representations?" I've also read the description of your question, and, unfortunately, I cannot answer it. However, I'd suggest that representations are always distributed, so the position information may be at a lower level than the identity representation. Counting visual objects, on the other hand, requires an eye-motion algorithm (except in the case of the acquired skill of subitizing) based partly on the skill of pure, abstract counting. Where's the result stored? – John Pick Jan 28 '16 at 19:12
• Thanks for pointing out. Also, it's a good insight that counting is a temporal process. And for very small numbers (that can be recognized at a glimpse) that might indeed be individual neurons representing the numbers. So the problem is more about positions. I have the impression that positions are very temporal, too. Do you know if the "face" neuron would fire only if there is a face right in front of you, i.e. in the center of the field of view where we have sharp vision? – danijar Jan 28 '16 at 19:18
• I'd expect the "face" neuron to fire if you were thinking of the appearance of a face, whether you could actually see it or not. But, I think you are wondering if the "face" neuron can fire when a face is detected only in periphery vision (and there were no other clues such as a voice or an attached body that would evoke "face"). I guess it would depend on degree of eccentricity of gaze, whether that part of the periphery has high enough resolution, and whether you have enough experience recognizing faces in that part of your periphery. – John Pick Jan 29 '16 at 5:15
• My question was not if periphery vision is enough to recognize faces. I though that maybe by design, the "face" neuron only fires when there is a face in the center of view span. This would explain how two faces would be represented, that is, they won't be represented at the same time at all. – danijar Jan 29 '16 at 18:05

Two ideas on this so far:

• I think we have neurons representing multiple occurrences of a given feature, for example one neuron for "one face", one for "two faces", etc. At some number it doesn't really make a big difference anymore, so there is just a neuron for "group of faces". This would explain why we can recognize small number of objects in a glimpse but have to count for larger amount. Corrections and suggestions are welcome.
• In his 2014 talk at MIT, Prof. Hinton explains how the brain might find equivariant features instead of invariant features. Basically, in addition to existence of a feature, neurons in the same cortical cell might output the properties of this feature. In vision, those properties would be position and rotation, for example.

You are basically asking how to bind different concepts together based off their representation in neurons. The one way I know how to do this is using the Semantic Pointer Architecture (SPA).

To understand how lower level neural activities can be compressed as a concept, please see my answer on compression in the brain using the SPA. This answer explains how low-level features can be compressed into a higher level concept vector.

Once you've accepted, that concepts can be represented as high dimensional vectors and these vectors can be manipulated using neurons, we now have to define what types of manipulations can be used to represent counts.

A perfect example of this is Dan Rasmussen's neural system for solving Raven's Progressive Matrices. In the system, graphical cells in a matrice are represent using a neural symbols bound together using circular convolution. For example, the top-left cell in the below image could be represented as symbols bound together as SHAPE*CIRCLE+COLOR*BLACK+NUMBER*ONE where every word written in ALL CAPS is a symbol represented by a high dimensional vector and based on sensory input.

To relate numbers to each other, the number ONE is defined as a unitary vector which can be convolved with itself to represent new numbers. For example TWO=ONE*ONE and THREE=ONE*ONE*ONE=TWO*ONE.

For implementation details and to try out these concepts for yourself if a simulator, please see Nengo and "How to Build a Brain" by Chris Eliasmith.

• Interesting mechanic. Would that also account for multiple types of objects and their positions and counts (assuming the brain can do that which is not clear)? – danijar Mar 7 '16 at 21:34
• The example I gave only covered the case of Raven Progressive Matrice, but hypothetically any structure is possible given a high enough dimensional vector. This raises two sub-questions: 1. What is the limit of the dimensions that the brain can represent? 2. How can we define this in a general manner in the vision system. For the first question, this is discusses somewhere in "How to Build a Brain", but it was a pretty large number, like 4096 if I'm not mistaken. For the second question, it depends on the vision architecture. – Seanny123 Mar 8 '16 at 0:37
• It's easy to imagine changing the receptive fields of a spiking version of a convolutional neural net depending on the task so that various structures can be extracted. However, implementing it is non-trivial. – Seanny123 Mar 8 '16 at 0:39