One of the fundamental operations in image neural networks is the operation of convolution, where a filter slides across the receptive field, activating on particular features. This is incredibly useful because it allows us to recycle filters, without having to train neurons on each piece of the receptive field.

It seems like such an operation is mechanically impossible within the human visual cortex. From the paper How does the brain solve visual object recognition - DiCarlo, et. al., the brain opts to instead use a massive amount of neurons in each layer, most of which are redundant in that many neurons detect the same feature at different locations. This is clearly unscalable to apply everywhere, so only about 10% of the visual field concentrates these neurons.

On the other hand, we do have the ability to perform convolutions by moving our eyes. So in effect, we can jitter our eyes thereby applying the same neurons to different parts of the scene infront of us. I would think that this, combined with short-term memory can help us process images.

Suppose I force a subject to focus their eyes on the center of some common object (e.g. a car), such that the object is within the center 10% of our visual field. Are there situations where the brain is unable to process what the object is, without jittering our eyes? Said another way, how much visual processing are we able to do without any physical movement of our eyes or head?

I'm interested in understanding this question because I'm not convinced that even the massive amount of neurons in the visual cortex are sufficient for visually parsing the object. Whereas by (even slightly) shifting the object in our vision, the combined featurizations from each location would significantly reduce the computational complexity.

  • $\begingroup$ The massively parallel convolution is only true for the simplest visual features which are important for detecting objects of all types (like edges). Cells with more specific best-stimuli always have bigger receptive fields. $\endgroup$ – Bryan Krause Apr 30 '18 at 20:05
  • $\begingroup$ @BryanKrause: But I think the "bigger receptive field" here is still around 10 degrees of vision (for example see Figure 4C on page 33 of the above link). That particular IT neuron is responding to binoculars and an otter, but not a face. It's still not clear what features exactly that neuron is capturing, but they are surely higher order combinations of lower-level features. $\endgroup$ – Alex R. Apr 30 '18 at 21:22
  • $\begingroup$ This is a really good question, that I wish would receive a full answer. (Came here to ask basically the same thing.) Introductions to convolutional (artificial) neural networks seem to always gloss over the fact that the weights are shared across the visual field in the computation, but that in the biologically-inspired structure (cat visual cortexes) I can't see any biological way for the neurons in different parts of the visual field to share weights the way we do computationally, apart from maybe @AlexR.'s notion of "jitter". $\endgroup$ – sh37211 Jul 28 '20 at 23:05
  • $\begingroup$ As far as I can tell, convolution was introduced by Fukushima in 1980 not because it was biologically-inspired, but rather because it allowed the artificial network to be less sensitive to changes in position. (His biologically-inspired part was the hierarchical arrangement of the layers.) ...but I'm kind of "asking" here, not "telling"! ;-) $\endgroup$ – sh37211 Jul 28 '20 at 23:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.