Biological plausibility of RBMs

Question

How biologically plausible are Restricted Bolztmann Machines (RBMs) and their stacked equivalent Deep Belief Networks (DBNs)? I know that Deep Learning (DL) in general isn't considered biologically plausible, since it requires information to be propagated backwards through uni-directional synapses. However, from what I understand, DBNs don't use back-prop and instead uses some sort of unsupervised learning algorithm. Does that make them more biologically plausible?

Answer 1

First, I want to clarify a few things. Deep Learning simply refers to any learning on a deep (more than one hidden layer) neural network, where the learning happens (i.e. parameters are adjusted) at all layers in the network. Therefore, deep learning algorithms can span the gamut between quite biologically plausible to very biologically implausible, and can include supervised and unsupervised (and maybe even reinforcement-driven) algorithms. That said, the algorithms that are currently very successful in deep learning (e.g. backprop on a convnet), are not very bio-plausible.

As pointed out in the comments, bio-plausibility is difficult to quantify. What is helpful is pointing out what aspects of an algorithm are or are not bio-plausible, so that's what I'll try to do here.

One mark in favour of RBMs is that the error signals are local. The objective function for an RBM is completely defined by the two layers of the RBM, and this holds even when the RBM is embedded in a larger network.

However, I am reluctant to say that the weight update of an RBM is completely local, at least in their typical formulation. RBMs are typically trained using Contrastive Divergence (CD) [1]. After the forward pass (computing the hidden node activations from the inputs), CD requires recomputing the input node activations from the hiddens, and then recomputing the hidden node activations from these new input node activations. (This can happen many more times depending on the particular variant of CD, but has to happen at least once.) This is called Gibbs sampling, and it is unclear to me how this could happen in biology.

Another mark against RBMs is that they have tied weights: the feedback weights are the transpose of the feedforward weights. Some autoencoders avoid this problem by having untied weights, though this can also make them more difficult to train (the increased flexibility leads to more local minima).

A Deep Belief Network (DBN) is initialized by stacking a bunch of RBMs together. One can think of the first RBM as encoding the inputs (i.e. finding a probability distribution over the inputs), the second RBM as encoding the encoding created by the first, the third RBM encoding this second encoding, and so on. Creating the DBN sometimes involves "untying" the weights, so that there are now separate feedforward and feedback weights.

Once constructed, DBNs can be fine-tuned using a number of algorithms, some more bio-plausible than others. Here, fine tuning simply refers to adjusting all the parameters of the DBN together to minimize some global objective function, rather than using the local objective functions for each RBM (as was done when creating the network). One fine-tuning algorithm is the wake-sleep algorithm [1], an unsupervised learning algorithm that has two distinct phases [2]: A "wake" phase, where the network is run forward, and the feedback weights are adjusted based on these feedforward activations; A sleep phase, where the network is run in reverse (generatively), and the feedforward weights are adjusted based on these feedback activations. This algorithm is reasonably bio-plausible, since the weight updates can be determined locally, and feedforward and feedback weights are separate and are adjusted separately. Some of the papers on the wake-sleep algorithm suggest that it might map onto and explain the wake-sleep cycles of animals, however I do not know of any studies that provide evidence of this.

DBNs can also be fine-tuned using supervised learning algorithms, namely backprop. This of course brings all the bio-plausibility concerns associated with backprop.

So to summarize, and address your overall question, which I interpret as "Are RBMs and DBNs more bio-plausible than deep networks trained with backprop?": It's going to depend on how exactly they're trained. RBMs run synapses backwards just like backprop does, but propagating things backwards through unidirectional synapses is only one of the bio-plausibility concerns associated with backprop. Others include that backprop requires the derivatives of the hidden units at their activation points, and that backprop uses continuous values, not spikes (for a fuller list, see [3]). RBMs avoid these two concerns, however they add additional concerns, like how Gibbs sampling might happen in biology. Which one is "more" or "less" biologically plausible depends on which of these problems might be more easily solved in the brain. With regards to DBNs, I would say that the wake-sleep algorithm is more biologically plausible than backprop, in that the mechanisms in the brain seem more tailored to implement the wake-sleep algorithm than backprop.

That said, could a DBN trained with wake-sleep and a shallow classifier on top achieve the same results as a convnet trained with backprop, on a task like ILSVRC-2012 (ImageNet)? Certainly not with our current computational power and the current machine learning knowledge and tricks. And even though I don't expect a good bio-plausible learning algorithm to scale to large datasets right out of the box, it is important to keep in mind that whatever algorithms we want to attribute to the brain, they're going to have to scale.

[1] https://www.ncbi.nlm.nih.gov/pubmed/16764513 [2] https://en.wikipedia.org/wiki/Wake-sleep_algorithm [3] https://arxiv.org/abs/1502.04156v3