34
$\begingroup$

One of the reasons artificial neural net algorithms like cascade correlation (pdf) have been generating interest is because they start with a minimal topology (just input and output unit) and recruit new hidden units as learning progresses. The draw from cognitive science (especially a lot of developmental psychology work) is an analogy to neuorogenesis (in fact, you will see this analogy mentioned in most papers that use CC-NN).

However, the CC algorithm is not meant as a biologically reasonable algorithm (neither the weight update nor unit recruitment rules are local, for instance), So my question is:

Are there popular neural network algorithms that start with a minimal topology and recruit new hidden units, and update network weights in a neurobiologically plausible manner?

Related questions

How are newly created neurons recruited into existing networks?

$\endgroup$
7
  • $\begingroup$ I notice that this question was asked a while ago. I would be interested to hear anything you may have come up with since then. $\endgroup$ Apr 16, 2012 at 6:51
  • 1
    $\begingroup$ The Wikipedia link has been deleted: cs.iastate.edu/~honavar/fahlman.pdf. Might get more people answering on CompSci cstheory.stackexchange.com/questions/tagged/ne.neural-evol $\endgroup$
    – Chris S
    Apr 16, 2012 at 9:01
  • $\begingroup$ @ChrisS unfortunately this not on-topic on cstheory, and I would not want to ask it there. Thanks for catching the dead link! $\endgroup$ Apr 16, 2012 at 11:26
  • 1
    $\begingroup$ @MattMunson thanks for your answer. I was actually organizing an online reading group on neurogenesis with some colleagues so that we could start seriously thinking about some of these questions. Unfortunately everything got delayed because there were a few other projects ahead in the queue. If you are interested in the reading group then send me an email. $\endgroup$ Apr 17, 2012 at 13:54
  • $\begingroup$ FYI your link was broken, I removed it. Feel free to add in a better link if you so choose! $\endgroup$
    – Josh
    Jun 6, 2012 at 19:43

2 Answers 2

14
$\begingroup$

I don't know of any NN algorithms that match your definition entirely, and I have looked for them (previously and recently). Here are some papers that I think are close or in the direction that you are exploring.

Using theoretical models to analyze neural development (review)

An Instruction Language for Self-Construction in the Context of Neural Networks

Evolved neurogenesis and synaptogenesis for robotic control: the L-brain model

Modeling new neuron function: a history of using computational neuroscience to study adult neurogenesis (review)

The first two links are computational neuroscience papers that discuss relatively complex models of neurogenesis as it relates to neurodevelopment (rather than adult neurogenesis). Some of these models (inlcuding the second paper) do not even involve network activity, and, in their current form, likely none of them are capable or suited for solving practical problems. The third paper is probably closest to what you are looking for (but I don't have access), and the fourth is as its title suggests.

Many network models of neurogenesis focus on neurodevelopment. A fairly obvious conclusion seems to be that the reason for that is that neurogenesis' most important biological role is in neurodevelopment. I have thought alot about using neuroscience to derive NN algorithms that are both biologically plausible and functional, and have considered neurogenesis within that context. My present conclusion is that, outside of neuroevolution, neurogenesis is not currently an ideal focus for NN modeling, because its role in learning and computation seems to be mostly limited to a special case (the hippocampus) that is not well understood (despite its obvious importance).

In regards to cascade correlation, I suspect that a similar effect may be achieved in some biological NNs using only synaptogenesis and synaptic plasticity. Basically, if you have very many neurons and new learning is confined to a minimal number of synapses that are subsequently protected from future modification, then the effect might be the same as always confining new learning to newly-added neurons (as in CC-NN). Such a case would be consistent with these findings, for example. In such a model, it would not be biologically plausible, and perhaps not desirable, for each neuron to be connected with each neuron in the preceding and following layer, and thus a system for determining the pattern of exploratory connections would be required. To do that, one could draw from neurodevelopment models such as the above (in order to bias the targets of synaptogenesis, to seed initial connections of the network, or both) or, alternatively, try to derive algorithms that approximate observed connectivity patterns of biological networks.

$\endgroup$
1
$\begingroup$

I don't know of a particular network model for this job (so my answer will be an incomplete one), but I believe that any Hebbian learning based associative memory can easily be structured to simulate neurogenesis. These unsupervised networks are actually nonlinear dynamic systems that can be understood in terms of their phase spaces. Phase space is the realization of the dynamicity of the network in time. It contains (possibly) several stable states, called minima.

Consider an n-node network. When you add the n+1th node and connect it to the rest of the network via a certain number of connections, what you (hypothetically) see is that the phase space is not reshaped entirely; but it changes smoothly. An important thing to clarify (thanks to Artem Kaznatcheev), is that number n should have a considerably big value. Imagine the extreme case, where n is 1; then the entire structure will be gone.

I am not that good at its mathematics, but in principle, this allows neurogenesis to be simulated on this kind of networks that implement dynamic systems. One can develop algorithms that recruits or removes nodes and connections paralled to training.

How to add nodes: I believe that the answer to this particular question does not effect the neurobiological plausibility of the model. We can even assume that the nodes are there, but not used (due to prior unnecessity), and now we need them and recruit them.

But, as an end note, I must say that I have not developed such a such a metwork before, so I am unable to share my experiences here.

$\endgroup$
3
  • $\begingroup$ There is no reason to expect a smooth or continuous change to the phase-profile when you do a discrete operation like adding a neuron. Further, the whole difficulty of the question lies in HOW to add neurons in a biologically reasonable way, which your answer does not address. $\endgroup$ Feb 12, 2012 at 15:41
  • $\begingroup$ I will be editing my answer by adding: 1) number n should have a considerably big value. 2) no need to add neurons, just assume that they are already there. $\endgroup$ Feb 12, 2012 at 15:52
  • $\begingroup$ "and now we need them and recruit them" the whole point of my question is to answer HOW to recruit them in a biologically plausible manner. $\endgroup$ Feb 12, 2012 at 18:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.