0
$\begingroup$

in this book Computational Cognitive Neuroscience and its corresponding software leabra

The Leabra Algorithm: Leabra stands for Local, Error-driven and Associative, Biologically Realistic Algorithm, and it implements a balance between error-driven (backpropagation) and associative (Hebbian) learning on top of a biologically-based point-neuron activation function with inhibitory competition dynamics (either via inhibitory interneurons or an approximation thereof), which produce k-Winners-Take-All (kWTA) sparse distributed representations.

a biological neural network is composed of 3 hierarchies: neuron -> layer -> network

Why is it not: neuron -> network ?

suppose you have a brain with $10$ neurons, then the neural network can be described as a $10\times 10$ connectivity matrix, and that's all, right?

Why do you need, for example, to divide the $10$ neurons into $2$ layers, and each layer has $5$ neurons, and how do you even distinguish them? (how do you know which neuron belongs to which layer?)

$\endgroup$
3
  • $\begingroup$ I see you've changed the title. Nonetheless, you link to an algorithmic implementation of a biologically-inspired network. They are describing their software. "Biologically-based" I think is throwing you off here, they are still talking about an artificial network. They have chosen to divide their network into layers, because they chose to do it, so that's what their software does. $\endgroup$
    – Bryan Krause
    Sep 7, 2022 at 16:41
  • $\begingroup$ You can read about biological layers in the Wikipedia page for Neocortex that I linked in my answer, but you will not find that these biological (anatomical) layers have very much to do with the layers in the model you link to. $\endgroup$
    – Bryan Krause
    Sep 7, 2022 at 16:42
  • $\begingroup$ It's usually called full Boltzman machine without layers in your case which is known to be of exponential computational complexity, usually one needs to add some layers to restrict complexity to be practical. Of course you need much experience and knowledge to optimally decide number of layers and number of neurons in each layer for a specific problem, usually more layers are preferred for difficult multi-dimensional input. $\endgroup$ Nov 3, 2022 at 2:40

1 Answer 1

2
$\begingroup$

This is not really a question related much to biological neural networks, rather it's a concept from artificial neural networks that are arranged in layered structure.

In a layered artificial neural network, "neurons" of one layer receive input only from the previous layer, and give output only to the subsequent layer. This creates a computationally efficient feed-forward structure: there's no complicated recurrence, once you know the activities of one layer you can calculate the activities of the next layer by a single multiplication step between the weight matrix and activity vector of the previous layer, followed by a sum for each unit.

From the README of the software you link to:

There are 3 main levels of structure: Network, Layer and Prjn (projection). The network calls methods on its Layers, and Layers iterate over both Neuron data structures (which have only a minimal set of methods) and the Prjns, to implement the relevant computations. The Prjn fully manages everything about a projection of connectivity between two layers, including the full list of Syanpse elements in the connection. ... The Layer also has a set of Pool elements, one for each level at which inhibition is computed

So, while the neurons in this network may have some biological aspects like integration over time, it seems they are arranged into layers like a classical ANN, information is shared between layers through a "Prjn" object that manages the "synapses" between two layers. Additionally, each layer seems to have its own inhibitory unit. I assume the way this is modeled is that every neuron in a layer gives input to an inhibitory unit, which then reduces the activity within that layer as sort of a gain control. Again, while they may be biologically motivated, these are all artificial constructions for a computational model chosen by the authors.

While you could conceivably still map these connections all into an NxN matrix, everything would be zero except for upper triangular matrices corresponding to each pair of layers. Nothing about hierarchical organization says you can't put it in a larger matrix, just that it doesn't necessarily make sense to do so. For example, you could put students in a classroom into a longer list by school, that doesn't mean there isn't an underlying organization of students within classroom just because you abstracted out that level.

While there are also layered structures in biological networks (like neocortex) these layers do not function like artificial neural network layers, as neurons within layers are strongly connected to each other, and while there is some evidence for a "feed-forward" pathway (roughly, layer 4 to 2/3 to 5 to 6), this is only a moderate bias in connectivity and only a minority of connections actually follow this canonical pattern.

$\endgroup$
3
  • $\begingroup$ i am asking about biological networks! the book Computational Cognitive Neuroscience and the leabra software are both doing biological simulations, not AI/ML stuff $\endgroup$ Sep 7, 2022 at 15:29
  • $\begingroup$ @DingRuiqi From what I see from the software, it's at best biologically inspired. They're choosing to organize things in layers, like typical ANNs. They're describing their computational architecture when they say it's organized in layers. They may not be using the strict feedforward network I described, but it's still an organizational choice they're making. $\endgroup$
    – Bryan Krause
    Sep 7, 2022 at 15:35
  • $\begingroup$ For example copied from the README: "There are 3 main levels of structure: Network, Layer and Prjn (projection). The network calls methods on its Layers, and Layers iterate over both Neuron data structures (which have only a minimal set of methods) and the Prjns, to implement the relevant computations. The Prjn fully manages everything about a projection of connectivity between two layers, including the full list of Syanpse elements in the connection. ... The Layer also has a set of Pool elements, one for each level at which inhibition is computed " $\endgroup$
    – Bryan Krause
    Sep 7, 2022 at 15:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.