Reverse inference, i.e. the use of brain activation (fMRI) data to infer the engagement of a certain mental function, is heavily criticised (e.g., Poldrack 2011, Neuron).

At the same time, decoding, the process of applying machine learning algorithms to such brain activation patterns in order to infer something about the mental function that elicited this response (e.g. to distinguish between two different conditions in an fMRI experiment), is extensively used and actively developed - multivariate pattern analysis (MVPA) being just the most widespread of these techniques.

The two concepts appear, however, to be one and the same; or rather, that the second (decoding) is a special case of the first (reverse inference). Is there an additional nuance to their difference? Also, why is decoding not criticised in the same way as reverse inference?

  • $\begingroup$ any thoughts at all, anyone? $\endgroup$ – z8080 Jan 4 '17 at 10:50

Short answer: Decoding is not a special case of reverse inference.

The difficulty with interpreting neuroimaging results is that there is a tremendous amount of variability (noise) in the data. For example, say we attempt to determine the brain areas associated with the emotion of romantic love by showing subjects images of close friends (condition 1), or images of their loved ones (condition 2), and comparing the results. Each brain scan may show 5-10 active regions, which regions are active and to what degree varies between subjects even in the same condition, and there is even variability in brain scans of the same subject across multiple trials.

To deal with this variability, the first step in just about any neuroimaging experiment's data interpretation process is a statistical analysis. This can range from an "averaging" or "noise-cancellation" analysis, to a multi-voxel / multi-frame machine-learning pattern-matching classifier (MVPA). The data analysis is used to determine a predictor of the independent variable, and also to calculate significance (p-value) based on the consistency level in the data. For example, we may learn that subjects viewing images of their loved ones (condition 2) have 5 areas of their brain reduce activity and 4 areas increase activity relative to subjects viewing images of other close relations (condition 1). Note that the decoding step does not involve "labelling" the pattern as any particular mental state - a classifier like MVPA does not care what the patterns mean, it's just a mathematical function used to distinguish between them.

enter image description here

In a typical cognitive neuroscience experiment, the next step is to attach a label to the mental state defined by the pattern found - this is forward inference. To do this, researchers typically determine the mental state through some other validated measure - for example, a questionnaire or some other test that is known to reliably measure love. The researchers may conclude: "These regions of the brain are associated with the emotion of romantic love when subjects view images of their loved ones." Immediately, a constraining factor arises: This pattern of activation is relative to the control condition only!

While forward inference involves labelling the pattern from the independent variable, reverse inference is used to label the independent variable from an activity pattern. Reverse inference gets into trouble when it is used to label a particular pattern of activation in a different context. For example, "the same areas of the brain associated with the emotion of love in an experiment involving showing subjects images of their loved ones, also show the same pattern of activation when subjects view their iPhones, and therefore we conclude that subjects are in love with their iPhones." Note that statistical analysis of the data (decoding) must take place first in order to be able to make either kind of cognitive inference.

Reverse inferences suffer from several potential pitfalls, among them:

  • A particular pattern of activity usually indicates different mental states in different contexts, so assuming that the same pattern indicates "love" when applied to iPhones is invalid.
  • Patterns isolated using a statistical analysis usually do not fully describe the mental state, so assuming that the pattern is sufficient to indicate "love" is also invalid.
  • Mental state labels themselves are context dependent, subject-dependent, and even temporally variable, so assuming that subjects would interpret their own mental state as "love" is invalid as well.
  • Mental states are not discrete, as evidenced by all the "noise" in neuroimaging data, so subjects may identify their mental state using a variety of different labels depending on what part of their mental state they attend to, "love" being only one.

Reverse inference is subject to these issues no matter what statistical analysis is used, MVPA is not immune. However, many of these issues can be overcome using a variety of techniques. MVPA is very valuable for dealing with the first issue - thanks to much higher "resolution" (level of detail, scope of data, linearity, etc), this decoding technique is significantly less likely to confuse different mental states that appear to have the same activity pattern at a lower resolution. Another important strategy is using meta-analysis (data gathered from many different studies) to determine how useful a particular pattern is for reliably inferring a mental state. Solving the remaining problems will likely require per-subject training data, and even more advanced analysis techniques, but already reverse inference has been a useful methodology in many studies, when carefully applied.

An additional interesting side-effect of machine-learning pattern analyzers is that they tend to remove the temptation to make unwarranted reverse inferences. In the early days, when patterns of activation were "informal" - based on gross anatomy - it was too easy to make sweeping statements such as: "... the insular cortex of the brain, which is associated with feelings of love ..." With MVPA, patterns of activation are practically impossible to describe in human language, so machines are required to make inferences, which are then inherently more objective.

  • $\begingroup$ Thank you, a very thought-through and helpful answer, though I am not convinced of your starting statement, that reverse inference is a special case of decoding. $\endgroup$ – z8080 Jan 11 '17 at 12:13
  • 1
    $\begingroup$ The way I am beginning to understand it now, reverse inference is similar to anecdotical evidence that makes hard-to-verify claims whereas machine-learning is more restrained in its labelling of the mental state that has caused the obsered brain activity, albeit having done a much more systematic analysis of it. Thus, while being in a better position to do reverse inference, the latter need not necessarily apply the label that I suppose all the Poldrack-related criticism attacks. $\endgroup$ – z8080 Jan 11 '17 at 12:13
  • $\begingroup$ One helpful reminder comes from the very paper I mentioned in the OP, Poldrack 2011: "Whereas the kind of reverse inference described above is informal, in the sense that it is based on the researcher’s knowledge of associations between activation and mental functions, a more recent approach [MVPA] provides the ability to formally test the ability to infer mental states from neuroimaging data." $\endgroup$ – z8080 Jan 11 '17 at 12:14
  • $\begingroup$ Also, you say that forward inference is to "attach a label to the mental state defined by the pattern found" - but that sounds like reverse inference to me, i.e. state <- data. Forward inference as I understand it is the opposite i.e. state -> data, i.e. the description of the activation pattern itself. $\endgroup$ – z8080 Jan 11 '17 at 12:35
  • $\begingroup$ @z8080 Lots of good questions :-) I can see that this is a broad topic that is difficult to cover in a single answer... I'll add some comments, and update my answer as needed to clarify. $\endgroup$ – Arnon Weinberg Jan 11 '17 at 19:52

Note. I initially scan read the question, I have rewritten my answer as a consequence, and due to the comments given.

As highlighted by others here Multi-voxel pattern analysis (MVPA) is an application of machine learning, used for decoding vast quantities of complex information (neural activation patterns to particular asks). This is a form of decoding may be used to infer a cognition, otherwise known as reverse inference.

The problem of reverse inference is largely summed up by the following comment from Poldrak's (2011)...

The use of reasoning from activation to mental functions, known as “reverse inference”, has been previously criticized on the basis that it does not take into account how selectively the area is activated by the mental process in question.

Poldrack (2011) goes on to explain that informal reverse inference, which is based on a researchers knowledge, is flawed because an individuals knowledge is limited by what the remember and have read. Additionally poor interpretations are compounded from one researcher to another.

The problem with reverse inference comes not from understanding general cognitive processes, such as sight, movement, language, decision-making etc. General patterns for general processes have been established, the problem as Poldrack (2011) points out is when we interpret patterns to more well defined cognitions, for example instead of looking at merely reward processing, we might want to compare patterns of pleasure derived from seeing highly palatable e.g. cake, and less palatable food e.g. fruit. At this level, comparing the data requires far more specific analytical approach. If we were a researcher making an inference based on our knowledge, we would have a strong chance of error.

MVPA deals with a far higher resolution of data than an individual could deal with, and compares data to prior trials or previous experiments, see Fig 1. However it is critical to remember that we are comparing participants within similar contexts.

Fig 1. MVPA diagram of testing and inference Norman et al (2006) Fig 1. MVPA diagram of testing and inference (Norman et al 2006)

Poldrack (2011) gives an excellent example of how to use MVPA, which was conducted by Kay et al (2008). Simply Kay et al (2008) scanned participants viewing natural images n= 1750. In the following trial 120 images were added, MVPA of the neural data was able to accurately predict which images were being viewed. This method has seen substantial development to the point that using a similar method researchers have accurately analysed neural patterns observed in a participants waking brain into decoding what participants are dreaming. Basically the quantity and quality of when taking account of context may be used to conduct reverse inference, but this does require previous data, within context, for the MPVA and other machine learning methods to compare too.

This does not mean that machine learning can not be wrong if incorrectly applied. It is a statistical method in which humans set the parameters, the post below by Arnon Weinberg accurately defines the issues and pot holes that must be avoided to make this method viable for reverse inference.

Norman, K. A., Polyn, S. M., Detre, G. J., & Haxby, J. V. (2006). Beyond mind-reading: multi-voxel pattern analysis of fMRI data. Trends in Cognitive Sciences, 10(9), 424–430. http://doi.org/10.1016/j.tics.2006.07.005

  • 1
    $\begingroup$ If I may translate your ideas to one sentence: because in Machine Learning there is so much data that says A means B, there is a large chance that B means A? Is that correct? $\endgroup$ – Robin Kramer Jan 10 '17 at 17:57
  • 1
    $\begingroup$ That's how I interpret this answer as well, and is exactly the kind of thinking that Poldrack criticises, so I don't think this answers the question correctly. $\endgroup$ – Arnon Weinberg Jan 10 '17 at 19:06
  • 1
    $\begingroup$ Thanks for your answer @Comte. Note that its very premise ("MVPA(..) is a form of reverse inference"), is opposite to that of Arnon's, which reads "reverse inference is a special case of decoding." $\endgroup$ – z8080 Jan 11 '17 at 12:16
  • 1
    $\begingroup$ Also, you say that MVPA compares the training pattern at hand with that observed in past experiments - however that is not what most MVPA papers report at all, unless of course they have a meta-analytical purpose. Can you link to a study that does that? $\endgroup$ – z8080 Jan 11 '17 at 12:20
  • 1
    $\begingroup$ @ Comte Good answer, but I wouldn't say MVPA is a form of reverse inference; reverse inference may or may not follow MVPA. @z8080 You are right that few MVPA studies make use of past data - Poldrack encourages researchers to consider doing more of that before making reverse inferences. $\endgroup$ – Arnon Weinberg Jan 11 '17 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.