Reverse inference, i.e. the use of brain activation (fMRI) data to infer the engagement of a certain mental function, is heavily criticised (e.g., Poldrack 2011, Neuron).

At the same time, decoding, the process of applying machine learning algorithms to such brain activation patterns in order to infer something about the mental function that elicited this response (e.g. to distinguish between two different conditions in an fMRI experiment), is extensively used and actively developed - multivariate pattern analysis (MVPA) being just the most widespread of these techniques.

The two concepts appear, however, to be one and the same; or rather, that the second (decoding) is a special case of the first (reverse inference). Is there an additional nuance to their difference? Also, why is decoding not criticised in the same way as reverse inference?

  • $\begingroup$ any thoughts at all, anyone? $\endgroup$
    – z8080
    Jan 4, 2017 at 10:50

2 Answers 2


Short answer: Decoding is not a special case of reverse inference.

The difficulty with interpreting neuroimaging results is that there is a tremendous amount of variability (noise) in the data. For example, say we attempt to determine the brain areas associated with the emotion of romantic love by showing subjects images of close friends (condition 1), or images of their loved ones (condition 2), and comparing the results. Each brain scan may show 5-10 active regions, which regions are active and to what degree varies between subjects even in the same condition, and there is even variability in brain scans of the same subject across multiple trials.

To deal with this variability, the first step in just about any neuroimaging experiment's data interpretation process is a statistical analysis. This can range from an "averaging" or "noise-cancellation" analysis, to a multi-voxel / multi-frame machine-learning pattern-matching classifier (MVPA). The data analysis is used to determine a predictor of the independent variable, and also to calculate significance (p-value) based on the consistency level in the data. For example, we may learn that subjects viewing images of their loved ones (condition 2) have 5 areas of their brain reduce activity and 4 areas increase activity relative to subjects viewing images of other close relations (condition 1). Note that the decoding step does not involve "labelling" the pattern as any particular mental state - a classifier like MVPA does not care what the patterns mean, it's just a mathematical function used to distinguish between them.

enter image description here

In a typical cognitive neuroscience experiment, the next step is to attach a label to the mental state defined by the pattern found - this is forward inference. To do this, researchers typically determine the mental state through some other validated measure - for example, a questionnaire or some other test that is known to reliably measure love. The researchers may conclude: "These regions of the brain are associated with the emotion of romantic love when subjects view images of their loved ones." Immediately, a constraining factor arises: This pattern of activation is relative to the control condition only!

While forward inference involves labelling the pattern from the independent variable, reverse inference is used to label the independent variable from an activity pattern. Reverse inference gets into trouble when it is used to label a particular pattern of activation in a different context. For example, "the same areas of the brain associated with the emotion of love in an experiment involving showing subjects images of their loved ones, also show the same pattern of activation when subjects view their iPhones, and therefore we conclude that subjects are in love with their iPhones." Note that statistical analysis of the data (decoding) must take place first in order to be able to make either kind of cognitive inference.

Reverse inferences suffer from several potential pitfalls, among them:

  • A particular pattern of activity usually indicates different mental states in different contexts, so assuming that the same pattern indicates "love" when applied to iPhones is invalid.
  • Patterns isolated using a statistical analysis usually do not fully describe the mental state, so assuming that the pattern is sufficient to indicate "love" is also invalid.
  • Mental state labels themselves are context dependent, subject-dependent, and even temporally variable, so assuming that subjects would interpret their own mental state as "love" is invalid as well.
  • Mental states are not discrete, as evidenced by all the "noise" in neuroimaging data, so subjects may identify their mental state using a variety of different labels depending on what part of their mental state they attend to, "love" being only one.

Reverse inference is subject to these issues no matter what statistical analysis is used, MVPA is not immune. However, many of these issues can be overcome using a variety of techniques. MVPA is very valuable for dealing with the first issue - thanks to much higher "resolution" (level of detail, scope of data, linearity, etc), this decoding technique is significantly less likely to confuse different mental states that appear to have the same activity pattern at a lower resolution. Another important strategy is using meta-analysis (data gathered from many different studies) to determine how useful a particular pattern is for reliably inferring a mental state. Solving the remaining problems will likely require per-subject training data, and even more advanced analysis techniques, but already reverse inference has been a useful methodology in many studies, when carefully applied.

An additional interesting side-effect of machine-learning pattern analyzers is that they tend to remove the temptation to make unwarranted reverse inferences. In the early days, when patterns of activation were "informal" - based on gross anatomy - it was too easy to make sweeping statements such as: "... the insular cortex of the brain, which is associated with feelings of love ..." With MVPA, patterns of activation are practically impossible to describe in human language, so machines are required to make inferences, which are then inherently more objective.

  • $\begingroup$ THanks for that. Although your answer overall is very helpful, I think one consistent source of confusion in it is that you refer to both forward and backward inference as "decoding", whereas really what forward models do is encoding. $\endgroup$
    – z8080
    Jan 12, 2017 at 10:28
  • $\begingroup$ As far as I understand given multiple sources, the unknown in a forward/encoding/generative scenario is the different weights that are to be assigned to the different regressors one has defined in their design matrix; and in a backward/decoding scenario, it is the design matrix itself, namely the specific condition that corresponds to a given pattern of brain activity. $\endgroup$
    – z8080
    Jan 12, 2017 at 10:30
  • $\begingroup$ And to sum up, the last paragraph of your answer in my view best addresses the original question I had. Thanks again for a great contribution. $\endgroup$
    – z8080
    Jan 12, 2017 at 10:35
  • $\begingroup$ @z8080 Well, I'm glad you found what you were looking for. I've edited the answer accordingly. However, note that decoding and encoding are complementary statistical methods (with decoding predicting the independent variable from activity patterns, and encoding doing the reverse), while forward and reverse inference are complementary cognitive methods (with forward inference labelling the pattern from the independent variable, and reverse doing the reverse). The statistical analysis is required before any cognitive inferences can be made. $\endgroup$
    – Arnon Weinberg
    Jan 12, 2017 at 18:44

Note. I initially scan read the question, I have rewritten my answer as a consequence, and due to the comments given.

As highlighted by others here Multi-voxel pattern analysis (MVPA) is an application of machine learning, used for decoding vast quantities of complex information (neural activation patterns to particular asks). This is a form of decoding may be used to infer a cognition, otherwise known as reverse inference.

The problem of reverse inference is largely summed up by the following comment from Poldrak's (2011)...

The use of reasoning from activation to mental functions, known as “reverse inference”, has been previously criticized on the basis that it does not take into account how selectively the area is activated by the mental process in question.

Poldrack (2011) goes on to explain that informal reverse inference, which is based on a researchers knowledge, is flawed because an individuals knowledge is limited by what the remember and have read. Additionally poor interpretations are compounded from one researcher to another.

The problem with reverse inference comes not from understanding general cognitive processes, such as sight, movement, language, decision-making etc. General patterns for general processes have been established, the problem as Poldrack (2011) points out is when we interpret patterns to more well defined cognitions, for example instead of looking at merely reward processing, we might want to compare patterns of pleasure derived from seeing highly palatable e.g. cake, and less palatable food e.g. fruit. At this level, comparing the data requires far more specific analytical approach. If we were a researcher making an inference based on our knowledge, we would have a strong chance of error.

MVPA deals with a far higher resolution of data than an individual could deal with, and compares data to prior trials or previous experiments, see Fig 1. However it is critical to remember that we are comparing participants within similar contexts.

Fig 1. MVPA diagram of testing and inference Norman et al (2006) Fig 1. MVPA diagram of testing and inference (Norman et al 2006)

Poldrack (2011) gives an excellent example of how to use MVPA, which was conducted by Kay et al (2008). Simply Kay et al (2008) scanned participants viewing natural images n= 1750. In the following trial 120 images were added, MVPA of the neural data was able to accurately predict which images were being viewed. This method has seen substantial development to the point that using a similar method researchers have accurately analysed neural patterns observed in a participants waking brain into decoding what participants are dreaming. Basically the quantity and quality of when taking account of context may be used to conduct reverse inference, but this does require previous data, within context, for the MPVA and other machine learning methods to compare too.

This does not mean that machine learning can not be wrong if incorrectly applied. It is a statistical method in which humans set the parameters, the post below by Arnon Weinberg accurately defines the issues and pot holes that must be avoided to make this method viable for reverse inference.

Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10(9), 424–430. http://doi.org/10.1016/j.tics.2006.07.005

  • 1
    $\begingroup$ If I may translate your ideas to one sentence: because in Machine Learning there is so much data that says A means B, there is a large chance that B means A? Is that correct? $\endgroup$ Jan 10, 2017 at 17:57
  • 1
    $\begingroup$ That's how I interpret this answer as well, and is exactly the kind of thinking that Poldrack criticises, so I don't think this answers the question correctly. $\endgroup$
    – Arnon Weinberg
    Jan 10, 2017 at 19:06
  • 1
    $\begingroup$ Thanks for your answer @Comte. Note that its very premise ("MVPA(..) is a form of reverse inference"), is opposite to that of Arnon's, which reads "reverse inference is a special case of decoding." $\endgroup$
    – z8080
    Jan 11, 2017 at 12:16
  • 1
    $\begingroup$ Also, you say that MVPA compares the training pattern at hand with that observed in past experiments - however that is not what most MVPA papers report at all, unless of course they have a meta-analytical purpose. Can you link to a study that does that? $\endgroup$
    – z8080
    Jan 11, 2017 at 12:20
  • 1
    $\begingroup$ @ Comte Good answer, but I wouldn't say MVPA is a form of reverse inference; reverse inference may or may not follow MVPA. @z8080 You are right that few MVPA studies make use of past data - Poldrack encourages researchers to consider doing more of that before making reverse inferences. $\endgroup$
    – Arnon Weinberg
    Jan 11, 2017 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.