1
$\begingroup$

So I close my eyes, turn around, open them for just a blink and close again. I have observed a chair. I can't tell you how the chair looks like in detail, but I can more or less tell its shape and color. What's more, if I'm not disturbed, I can remember that shape and color for at least a minute.

What if I wanted to construct a device that could repeat this process. With digital hardware its not too hard - take a digital camera and a solid state memory device. For every bit of info in my solid state drive, I can call the operation "store" and my bit will be stored, or "erase" and it will be erased. Don't know how reliably each physical bit works with current technology, but given sufficient number of physical bits per logical bit I can theoretically make a device that will store my picture for as long as I want.

What about building a spiking neuronal network to do the same thing? I have tried some naive approaches, and I arrive at a fundamental problem: spiking neuronal networks seem to have almost no inertia on the fast timescale. A neuron only fires while there is sufficient input. When input is gone, it no longer fires (besides some resting state activity). I can construct a network with recurrently-connected neurons, in which case, in principle, the activity of the network could sustain itself after the input is gone. However, depending on the parameters of the network I observe a rapid switch between two very extreme behaviours. In one case, the feedback strength of the network is too weak, and the activity drops as soon as the input is gone. In the other, the feedback weight is too strong, and the activity of the network quickly reaches some maximal firing state independently of which exact input the network has received. These transitions happen exponentially fast. Theoretically, there are adaptive control mechanisms in neuronal networks, such as inhibition and synaptic plasticity. However, I am struggling to understand how such mechanisms can ever be fast enough to compensate for the sudden increase or drop of input.

TL;DR: Naively neuronal networks do not seem stable enough to implement robust short-term memory. What are the known mechanisms that nevertheless make it possible?

Note: I'm not asking how exactly short-term memory works. I am aware that nobody really knows. I'm asking for intuition to help disprove my argument that it should not work when implemented using neurons.

$\endgroup$
1
$\begingroup$

While we don't entirely know how all aspects of working memory operate, experiments show that a subset of cells in the brain exhibit persistent activity during working memory tasks.

Here's an example from a review by Curtis & D'Esposito:

enter image description here

This is a task in panel (a) where there is a cue presented briefly (marked by "C"), and then the monkey needs to remember the cue (during the time marked by "D"), and finally at time R the targets appear and the monkey can make a response.

We see in spiking recordings from monkeys that there are cells in dorsolateral prefrontal cortex that are activated not only by the cue and during the response, but they have persistent activity during the delay period. In human studies we can't ordinarily place electrodes but part (b) shows a similar study using fMRI, and the colored areas on the brain indicate brain areas that demonstrated more blood flow (marking metabolism) during the delay period, which includes areas of the same brain region as the monkey experiments.

The explanations are recurrent network activity and bistability: a population of cells that is excited can continue to excite each other and remain active until stopped (see for example attractor networks for a conceptual idea), as well as bistability in individual neurons where a neuron that reaches threshold continues to fire tonically rather than firing a single action potential, until firing is ceased via inhibition. Camperi and Wang 1998 is an excellent, clear review of these concepts with a straightforward mathematical approach to how you can model this in silico. It's probably more efficient to start from here rather than reinventing it on your own.

I also recommend other papers by XJ Wang who has done extensive work in this area. I also recommend you look at phase-space diagrams which can help conceptualize dynamic systems with stable properties (and which network parameters are necessary for stability), they come up a lot in theoretical neuroscience. Dayan and Abbott's Theoretical Neuroscience is also a good starting point if you can find a copy.


Camperi, M., & Wang, X. J. (1998). A model of visuospatial working memory in prefrontal cortex: recurrent network and cellular bistability. Journal of computational neuroscience, 5(4), 383-405.

Curtis, C. E., & D'Esposito, M. (2003). Persistent activity in the prefrontal cortex during working memory. Trends in cognitive sciences, 7(9), 415-423.

$\endgroup$
7
  • $\begingroup$ Thanks a lot for your reply! I was not aware of bistability. I have looked at several models of working memory before, but I have not seen anybody using it. It's not the same as bursting, is it? I'll read the literature by Wang you have recommended and get back to you. A small question if I may. The persistent activity in Esposito et al - it is persistent in every trial individually, or in trial-average picture? I have heard there is a large controversy between persistent and 'transient' activity, whatever the latter is $\endgroup$ Dec 21 '20 at 19:51
  • $\begingroup$ @AleksejsFomins It's a trial average. You can have persistent activity that's either relatively constant or persistent activity that comes in "waves"; this is true for sensory responses, too, see work by Ken Harris and Artur Luczak. $\endgroup$ Dec 21 '20 at 19:55
  • $\begingroup$ If you're coming at it from ANNs I'm not sure what the motivations would be for directly modeling working memory, if any. It's a biological solution, not necessarily relevant to artificial networks that have other tools available. $\endgroup$ Dec 21 '20 at 19:56
  • 1
    $\begingroup$ No no, I'm something in-between. I'm well aware that brain-inspired ANN have no chance of competing with digital hardware or known ML algorithms. I'm trying to solve credit assignment problem in the brain by means of reservoir/echo state networks. The idea is nice - reservoir does some generic space/time transform on incoming data, and a shallow readout can then learn whatever it needs. Bio-plausible and maybe in future can improve training cost of simple ANN. The main problem there is that most implementations to date forget input very fast, which makes them useless. Need to tune $\endgroup$ Dec 21 '20 at 23:09
  • 1
    $\begingroup$ @AleksejsFomins Yes, if you propose some second messenger increases intracellularly when a spike fires (or a synapse is activated) and decays with some (relatively slow) time constant you need not model all the intracellular metabolic processes. In the biology you expect some nonlinearities but depending on your goals those may not be important (or, at least, can be compromised away). $\endgroup$ Dec 21 '20 at 23:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.