What role do circular network structures play in neural networks?

Question

I'm developing small virtual neural networks and for the most part, the networks have remained feed-forward. When I create circular structures, it becomes a challenge to find a balance with the neural refactory period, stopping neurons from firing again for a period of time, giving them high action potential thresholds, then reducing that back down to fire again -it's tough to find a balance that stops infinite feedback loops but allows circular structures to play a role in processing.

I've done some reading on this and my vague idea so far is that circular structures perhaps allow the network to react to information based on its pattern, react to the information over time, so the C in an input series of A, B, C can be handled differently than the C in input series A, A, C.

Is that accurate, basically? What's the role of circular connection structures in a neural network? I'm asking not only about multi-neuron circular structures like N1 => N2 => N3 => N1 but also about neurons that connect to themselves (N1 => N1) (which from my reading is something that happens if I understand correctly, but I'm unsure whether it achieves a similar affect as a larger scale circular structure).

_{(=> represents a neural connection)}

Answer 1

In contrast to artificial neural networks, which are almost all feed-forward architectures, networks in biological brains are highly recurrent. In the networks of cortex, the majority of synaptic inputs a neuron receives come from nearby neurons in the same area of cortex (Binzegger et al. 2004). These are the "circular" connections that you are referring to.

So recurrent connectivity is the rule, and simple feed-forward (or feed-back) projections between cohorts are the exception. The networks that result are therefore highly dynamic, and respond in complex ways to inputs changing over time. In general there is no one "reason" why recurrent connections exist. But there are guidelines about how connections between excitatory and inhibitory neurons are made.

The patterns of connections which define a network determine the computational properties of that network. At a first level, the strength and connectivity between excitatory and inhibitory neurons control the overall stability of the network (e.g. Wilson & Cowan 1973; Tsodyks et al. 1997). It's certainly possible to have self-connections and high recurrence, and still keep the network stable.

Secondly, changing the rules by which neurons connect to each other (Ko et al. 2011; Cossell et al. 2015; Lee et al. 2016; Wilson et al. 2017) changes the computational properties of the resulting network (Muir & Mrsic-Flogel 2015).

In general, biological neural networks are highly recurrent dynamical systems which bear only passing relationship to the acyclic networks that we use to think about feed-forward and feed-back connections in the brain.

Answer 2

I'm not sure if this offers you any help, but I'll give it a shot.

I work with certain programs that allow me to create virtual networks. What these programs have in common is that they fire neurons one by one. So if you have a perceptron with size {input: 1, hidden: 5; output: 1} it will activate the input, then it will activate the hidden neurons chronologically and then the output.

For more complex feed-forward networks, neurons still get activated one by one using Topological sorting.

---

But in circular neural networks, or networks which have connections going backward, neurons still have to be activated one by one. So say I have the following neural network.

So the order of activation in this neural network is in, in, a, b, c, d, e, out. But as you might see, there is an error pointing from d backwards to b.

So how can b be influended by d when b gets activated first? Because neurons store their activation.

So if I run this neural network the first time, the influence from d on b is 0, because d has no activation yet. But after this first activation, d gets a value. So next time you run the neural network, b will be influenced by the previous value of d!

So arrows pointing backwards are also called memory connections; they make the previous activation values have influence on the next activation.

I hope this is somewhat clear O.O. LSTM networks use these connections.

Answer 3

Since this question is in the Cognitive Science category, I'm assuming you're asking for ideas supported by neuroscience evidence, and not artificial neural networks. Naturally, this makes things much more theoretical and difficult to get everyone on the same page.

...so the C in an input series of A, B, C can be handled differently than the C in input series A, A, C.

If I understand your example correctly, you are asking what the role of "circular" network connections is in processing series that vary over time, and with respect to the temporal context of the inputs. This is a very clever thing to ask and has many important applications to AI.

Conceptually, the brain does do this by having a mechanism to distinguish between inputs based on context. In your example, you are right in thinking that the C should be handled differently in ABC than it should in AAC. The ability to understand C in a way that is unique to the events leading up to it is a powerful one.

Contextual time series analysis is something that Jeff Hawkin's Hierarchical Temporal Memory (HTM) model of intelligence (based loosely on biologically inspired ideas) tries to accomplish, and it may be relevant to you. However, it does not incorporate feedback connections from higher (more abstract) regions to lower ones. Keep in mind that HTM still has its pitfalls in contextual time series analysis and so the actual way the brain does it is much more robust. I offer this merely as an anecdote.

From a biological perspective, you have to understand the word "circular" to mean the feedforward and feedback connections between neurons of different levels, as opposed to lateral connections between neurons within the same level. Neurons within the same level receive connections from the same inhibitory neurons (which are overwhelmingly local connections). You mentioned that neurons connect to themselves (N1 => N1), but this is not a very commonly studied thing in biology and probably has a role in error correction, boosting its own activity, or some other kind of activity modulation.

What seems to be happening in all parts of the brain, and even the retina, is that a level's excitatory neurons will send their feedforward signals to the next higher region's inhibitory and excitatory neurons. The inhibitory neurons will laterally inhibit the least active excitatory neurons within the same level. Some excitatory neurons will then send connections to the next higher region (feedforward) and the same process repeats. Other excitatory neurons will send connections to the lower region that connects to it (can consider these connections feedback), but will generally connect in a way to inhibit the excitatory neurons of the lower region by activating the inhibitory neurons (which work locally, thus only within that region).

With many more details excluded, this leads to a self-stabilizing framework where very little fine-tuning of arbitrary parameters is required (which apparently is a problem that you are well aware of in trying to find that balance). However, a circular network is just one substrate that is in use by the brain to understand context in time series. Unfortunately, circular connections actually have very little to do with contextual time series. Lateral connections seem to have much more to do with that than feedforward and feedback connections. With that said, there is much to be gained in combining the HTM's contextual time series analysis approach with a valid feedforward and feedback connection scheme between levels.

Feedforward connections mainly serve to make input signals much more abstract. Feedback connections serve as a way to fine-tune the lower levels and "tell" those levels which activations it is allowed to have.

If you want to learn more about circular connections and their role, I strongly recommend you study the retina. It is simpler than understanding the cortex and subcortical components, and we know much more about its cognitive processing than we do of the brain. Furthermore, the retina is actually an extension of the brain. It even comes complete with neurons that process signals in complex ways. You find the circular connections of which you speak in there, inhibitory/excitatory neurons, distinct hierarchical levels, and the higher cells even demonstrate characteristics of context-dependent input series on very small timescales.

Answer 4

The Neural Engineering Framework uses Principle 3 to describe these "circular" or "recurrent" connections as dynamical systems similar to those used in control theory. However, the only reason it's able to do this is it's focus on spiking neurons and modelling their dendrites as filtering, so I'm not sure how much this applies to your case, but I thought it might still be a helpful hint on a different approach if you're interested.