"A disturbingly large, and quite prominent, segment of social neuroscience research is using seriously defective research methods..."

This is one of the conclusions of an exciting paper (download here) first-authored by Edward Vul, in press in Perspectives on Psychological Science. It's a methodological critique of studies showing implausibly high correlations between brain responses and social behaviour. What's implausible is not the existence of the correlations - few people today claim that thought  proceseses are not localized somewhere in the brain. It's their sheer size. Given the margin of error that we can plausibly estimate for behavioral measures and measures based of brain activity, correlations should not reach a certain ceiling - yet, in social neuroscience, they do... After questioning the authors of 54 papers, Vul and colleagues have figured out that  in most of these studies, the correlations were due to a pre-selection of the data: in a nutshell, only those points of the brain scan where brain activity was significantly affected by a social variable, were used to compute the correlation. "This bias", the authors note, "was sometimes wielded selectively, in such a way as to inflate certain correlations, and not others".

The targets of the paper include such prominent papers as Tania Singer's much-celebrated Science paper on empathy and fairness, another Science paper by UCLA's social cognitive neuroscience lab, and many other leading figures in empathy research, such as Christian Keysers. It is a little upsetting to see the authors concentrate their attacks on social neuroscience and empathy research - maybe similar mistakes could be detected in other areas of neuroscience, although I am far too unqualified even to make guesses. But empathy research is not exactly new to accusations of scientific desultoriness (two words: mirror neurons).

Methodological mistakes are quite easy to correct - but they do tell us something important about the incredible success of social neuroscience: first, major reviews are ready to relax their standards in order to publish on topics that arise interest among the general public, such as men-women differences in cognition, empathy, fairness, love, etc. Second, there is something magic about brain-behaviour correlation. Although many readers of scientific reviews are, presumably, materialistic enough to admit that thought happens in the brain, many of them are still puzzled to find out that thought processes may actually correlate with brain activity. How else can we account for the popular success met by those studies that show changes in brain activity correlated with prayer, punishment, etc.?

Some reactions to Vul's paper are telling in this respect. These reactions typically take the following form: "Inflating correlations is bad science. But after all, what do we care if the correlations were inflated? What is fascinating about all these papers is that they observed any correlation at all! Isn't it amazing that social processes should have neural underpinnings?".

Well, no, it's not amazing if you believe that thought, including thought about social topics, does happen in the brain. Once you admit that, many (though by no means all) papers in social neuroscience lose much of their glamour. Correlation sizes, on the other hand, matter terribly - I'll let Vul et al. explain why:

"The magnitude, rather than the mere existence, of the correlation is what ‘really matters’.  A correlation of 0.96 (as in Sander et al., 2005), indicates that 92% of the variance in proneness to anxiety is predicted by the right cuneus response to angry speech.  A relationship of such strength would be a milestone in understanding of brain-behavior linkages, full of promise for potential diagnostic and therapeutic spin-offs.  In contrast, suppose (...) the true correlation in this case were 0.1, accounting for 1% of the variance. A correlation of 0.1 could be mediated by a wide variety of highly indirect relationships devoid of any generality or interest. For instance, proneness to anxiety may lead people to breathe faster, drink more coffee, or make slightly different choices in which lipids they ingest. (...)Or perhaps anxious people are more afraid than others of failing to follow task instructions and attend ever so slightly more to the required auditory stream.  The weaker the correlation, the greater the number of indirect and uninteresting causal chains that might be accounting for it, and the greater the chance that the effect itself will appear and disappear in different samples in a completely inscrutable fashion (e.g., if the dietary propensities of anxious people in England differ from those of anxious people in Japan)."

Anyway, the paper will certainly arouse embarrassment and controversy - all the more reasons to make your own opinion by reading it. And see also Mind Hacks and The Neurocritic.