Related/bonus points: I seem to remember reading about some equation that states the amount of information that can be held by a neural network with n neurons in it arranged in l layers, or something vaguely like that (n and l probably weren't even the letters in it.) Can anyone help me remember what I'm thinking of?

(The brain is a very large neural network. So, if we have an equation for neural networks, we should be able to get an estimate of the information contained in the human brain's neural network.)


Disclaimer: Quantifying the capacity of the human brain is quiet complex as you might imagine. And although in cognitive neuroscience we often compare the brain to computers this is not an exact comparison, in many ways the brain is far more complicated and encodes information in a very different way than the comparison of CPU processors and hard-drives. The short answer is that we have understandings about capacity under specific situations regarding STM, but LTM capacity is mostly based on estimates.

TL;DR: Estimates vary from 10^13 to 10^18 bytes (10 terabytes - 1 exabyte), and even this range should probably be taken with a grain of salt.

The long version:

Memory in the brain is often split by cognitive psychologists into several different modules, such as the memory storage model suggested by atkinson and shiffrin; attention->short-term(STM)->long-term(LTM).

Fig 1. atkinson and shiffrin model of memory

This model is largely inaccurate as the brain encodes information relative the type of information received, for instance auditory inputs will be processed by neural areas associated with auditory processing first. In addition attentional processing, unlike in this model, relies heavy in precognitive processing e.g. if you are hungry you will attend more to food. That being said we still use STM and LTM to distinguish between memory that is in use and memory that is stored.

Short-term memory capacity

The rather brilliant researchers Baddley and Hitch developed what is perhaps the most compelling models for short-term memory processing. The working memory model (see Fig. 1) accounts for differences in information types and how it is held within the processing centres of the brain.

Fig 2. Working memory model

A meta-analysis of 400 studies has displayed good support for the three main modules of cognitive memory. Generally speaking the central executive can be considered as the processor, while while the episodic buffer, phonological loop and visit-spatial sketchpad may be the modular RAM which holds information for processing.

According to Baddley the phonological loop can hold about 2 seconds of auditory information, this would be a list of unrelated words with a task designed to restrict rehearsal and encoding of information. However if the information was related, say "Our lecturer told us to read chapter 3 of working memory" we could hold more of the information as its related and may be chunked together. 'Chunking' is a feature of memory that complies similar information together. The capacity recorded generally depends on numerous factors, such as the task type, time between learning and recall, and the significance of the information. In addition we can also add age and context (internal and external) as factors influencing memory recall. Overall we can't identify the exact capacity of working memory due to the complexity of information processing, what we can say is that working memory deals with small amounts of information spit across different modules, and relates this to LTM. Although that small amount of information is probably a lot more than your average super computer can process as this post indicates.

Long-term capacity

As with STM, LTM is modular in anyways. However the total capacity is somewhat related too neurons, with 86-100 million neurons and 1000 glia cells thats means the human brain has a large capacity for storing information. However as previously mentioned these neurons relate to particular types of information. With regard to processing capacity the human brain is estimated to be [10 to the power of 17] 11 flops per sec, according to the blue brain project. Another recent estimate puts the processing capacity of the brain at 10^28 Flops.

Fig 3. A comparison of recent predictions of neural processing capacity and current fastest Super CPU Recent comparison of predicted neural processing capacity and current fastest super computer

A different measure of calculating processing capacity has been devised by Grace and Christiano who stated the following...

We can use Traversed Edges Per Second (TEPS) to measure a computer’s ability to communicate information internally. We can also estimate the human brain’s communication performance in terms of TEPS, and use this to meaningfully compare brains to computers. We estimate that the human brain performs around 0.18 – 6.4 * 1014 TEPS. This is within an order of magnitude more than existing supercomputers. TEPS = synapse-spikes/second in the brain

= Number of synapses in the brain * Average spikes/second in synapses

= Number of synapses in the brain * Average spikes/second in neurons

= 1.8-3.2 x 10^14 * 0.1-2

= 0.18 – 6.4 * 10^14

So the brain operates at around 18-640 trillion TEPS, while closest supercomputer is 2.3 * 10^13 TEPS (23 trillion TEPS).

Memory has been calculated to be around 2.5 petabytes (2.5 * 10^15 bytes), as reported here (seems to be based upon speculation by Prof P Reber). Another estimate has the neural memory capacity at 8∙10^19 bits--that's over 8 quintillion (10^18) bytes. Some researchers at Berkeley have suggested a relatively small 10-100 terabytes (10^13 to 10^14 bytes). All these estimates are based on variations in calculations relative to neuron density and and synapse connections across the whole brain. The larger estimates take addition account of other factors involved in neural communication. But the overall criticism I have is that we can't simply say that one synapse is 1 byte or 200 calculations per second.

The term estimate is generous here; guesstimate would be far more accurate. Individual neurons are complicated enough; moving to distributed and interconnected networks in the brain is just another level. Neurons do not conduct calculations independently; they rely heaving on context and information types. So we can say the processing of the brain is modular, in fact we already know this to be true to initial sensory processing. There is also no clearly obvious separation of memory from processing, although we know in motivation for instance certain areas will activate (the so called 'reward system') when assessing motivational objects. We won't say this is memory but it relies on it previous associations in memory. So some areas of the brain are used to assess rather than recall but they will activate together. The point being that we can't bundle all neurons together to calculate memory or processing. We just don't know what the vast amount of neurons are doing right now.

I highly recommend this article, for more information and references regarding computational and memory estimates.

  • 1
    $\begingroup$ @gfdsal As I mention, it is an academic guestimate, the brain simply does not store or access information in the same way as hardware that much is clear from recall accuracy. Even when thinking of neural networks, even spiking, we are at best doing a crude interpretation of what we find biologically at this time. For example activation of neurons and patterns of activity to access information is more complex than one neuron simply activating another. Which is part of where the potential lies for additional storage and processing etc, while complex SSD are relatively simple by comparison. $\endgroup$ – Comte Jan 11 at 11:53
  • 1
    $\begingroup$ That is not to say SSD aren't complex, they clearly are in construction, but they are relatively simple in terms of functionality. This is part of their beauty for information storage and speed. But modelling and measuring neural activity isn't as clear cut as we often teach it: sciencedirect.com/science/article/pii/S0301008218300509#bib0015 and that's before we get into neural structures, local and global effects and signalling patterns etc etc, are modelling of neurons is, therefore, still far from perfect: journals.physiology.org/doi/full/10.1152/jn.00360.2016 $\endgroup$ – Comte Jan 11 at 12:24
  • 1
    $\begingroup$ @gfdsal I hope that helps, even if the answer (non-answer) is somewhat unsatisfactory. $\endgroup$ – Comte Jan 11 at 12:25
  • 1
    $\begingroup$ well i got the intuition of what you are saying that we shouldnt look into quantity comparison between a neuron and SSD's based on their capacity to hold charge/bit but we should focus on the way computation is realized (activity). So basically a small program written in c++ can take more space compared to perhaps a more optimized program for same "activity" written directly in assembly.. is this intuition right? $\endgroup$ – gfdsal Jan 11 at 14:01
  • 1
    $\begingroup$ @gfdsal That is probably a better analogy but in truth it is storage, processor, software/code and the robotics all at once, built in to a neural network, with variance in adaptability dependent to inputs, which are weighted and designed by genetics. These networks start their initial programming with some basic code with inefficiently structured hardware and lacking data, they then continuously 'train' this neural net causing it to reshape and rebuild the underlying code and the associated hardware itself within the limitations of the original design. $\endgroup$ – Comte Jan 12 at 18:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.