Monday 14 June 2010

On why fMRI is bullshit, even when you're doing it right

I'm just back from the 11th European Workshop on Ecological Psychology and I have a few posts about things that arose there. Ecological psychology is, reassuringly, in better empirical form than I had been thinking; there are still  people doing things badly, but plenty going after things as carefully as they should be, with due attention to issues of information. So that was good.

There was one talk by John Wann on the neural control of steering in driving tasks. I like John a lot, and respect his work - he's a very careful experimentalist and not at all susceptible to fads in psychology, and his work has always included detailed use of perceptual information in sensible ways. He has recently been involved in some fMRI versions of his steering studies with a post-doc trained in imaging, and he presented this data to the conference.

One of the things I like about John is the care with which he does things. I mentioned in my previous rant about fMRI that one of the big problems with these studies is that the researchers have no theory of information: they don't understand what they are pumping into the brain, and therefore can never actually interpret what happens in the brain as a result. John is the exact opposite of this; he uses optic flow displays rich in information but rigorously controllable, and as a result is able to run an fMRI study well.

The key to fMRI is comparison. You can't simply measure the brain activity at a given time, because even the simplest task makes most, if not all of the brain light up. What you can do is measure the activity under conditions X, then under conditions X+Y, then subtract one from the other to establish the neural correlates of Y. While this is essentially Donderian subtraction and subject to the same basic critique (that it assumes independent, non-overlapping processes and fails to the extent that this is violated) it is not necessarily a bad plan, and you can help refine your estimates by having other cases to compare, so as to triangulate the activity unique to Y.

If you don't have a theory of information, then you run the risk that what you pump in at X+Y is actually A+B; see my previous rant again for my tale of the biological motion perception study. In John's study, this is very much not the case. He is able to present observers with optic flow fields and add or subtract genuine sources of information, and this allows him to, at the very least, be running a valid study. The net result is some specific activation in an interesting part of the brain that can be tied to variation in behaviour during the task and related to the information content of the displays.

This was all fine, and I was very impressed: but I was left with one question (which, to be fair, is the one question John also tries to keep in mind too): so what? These displays comes from the detailed behavioural studies he and his colleagues have run over the years, and on the basis of those studies they have come up with a pretty solid model of what the relevant information is, how and way it's detected and how it's used to control steering (although it's not the only one; see Fajen & Warren, 2003). These differences in behaviour now have a neural correlate; but so what? Knowing where in the brain the relevant information was showing up doesn't seem to add anything at all to the account: they already knew what the information was, and that behaviour varied as a function of the accessibility of that information. The fact that a bit of the brain also behaved differently as a function of the accessibility of that information is, well, just not that surprising; after all, clearly the brain is involved in all this.

John, good scientist that he is, has asked himself this question and convinced himself that these data do add a piece to the overall puzzle, acting as supporting confirmatory evidence for the behavioural differences. He doesn't claim it's better evidence, or a more real effect, just that it's a part of the puzzle; and at one level I can't get too worried by that. And, to be fair, the data did potentially come in handy at a later talk looking about how patterns of brain activity in infants varied as they learned to understand what looming means (objects that are approaching you appear to 'loom', i.e. grow in apparent optical size) - one of the bits lighting up did seem related to the bit that John is now fairly sure is involved in the perception of precisely this kind of information in steering. If true, this uses data from a task where adults were able to respond in a meaningful way to interpret neural data from an infant who couldn't; it's inferential, but that can be ok. I wasn't that convinced, but the general idea is sound.

But all this has mostly just reaffirmed my view that fMRI just isn't worth the huge amounts of money it costs. Psychology is a field where adding expensive imaging in where it isn't needed is seen as a sensible way to help gain funding, and I'm more convinced than ever that it simply isn't helping. John's careful experimenting showed one thing: that he had already gained all the necessary data to produce a good model without the brain data, which, if it did anything, simply confirmed what he already had good reason to believe.

References

Fajen, B. R.,& Warren, W. H. (2003). Behavioral dynamics of steering, obstacle avoidance, and route selection. Journal of Experimental Psychology: Human Perception and Performance, 29, 343-362. DOI: 10.1037/0096-1523.29.2.343

Field, D.T., Wilkie, R.M., Wann, J.P. (2007).  Neural systems in the visual control of steering.  Journal of Neuroscience, 27,  8002– 8010. doi:10.1523/JNEUROSCI.2130-07.2007

6 comments:

  1. I will try and once again leap to the defense of neuroimaging...

    I am in general agreement with your stand on fMRI as a supplement to behavior. We found region x in the brain that is interested in the crucial manipulation in our task (say, direction of optic flow). So, this brain region corresponds with the change in behavior that we see with our manipulation. Sure, whoop-de-do. Some part of the brain is involved in this perceptual discrimination. Hey, perhaps its even interesting that this part of the brain is not in the back of the brain, where visual processes are thought to happen.

    Perhaps there are some clever conclusions that can be drawn from combo behav and neuroimaging studies - task A and task B are, on face value, quite similar. However, they recruit vastly different neural structures to be completed (the dorsal/ventral stream dissociations are a pretty good example of this).


    But still, not really a huge bang for your buck in terms of our understanding of what makes us tick, in terms of inputs and outputs. Nothing that couldn't be figured out with clever behav experiments.

    But this is solely from the behavioral point of view - as an ecological psychologist I can see why behavior is all that interests you. Behavior is really all that matters at the end of the day - the only 'outcome'.

    But a whole other class of researchers cares little about behavior directly - they only want to understand how the brain, as an organ 'works'. Investigating whether brain region X, which previous research has shown to be interested in a certain task (motion perception), also plays a role in more complex variants of the task (optic flow in depth). If so, then the 'purpose' of that region gets redefined. New light is shed upon the previously puzzling symptoms of patients with lesions including that particular part of the brain. Clever behav experiments are designed to pick seemingly identical behaviors apart. This sort of thing can also be accomplished by fairly brutal monkey work. But I suspect that monkeys are an even more expensive way to go about addressing your questions, and certainly not an alternative for those weak-of-stomach.

    None of this is, I suspect, the point you were making - you study a brand of psychology that, almost by definition, cares nothing about the underlying brain processes.

    And of course, this is your blog and you are at liberty to say whatever you feel like. But do you really believe that functional imaging is, when used appropriately, an extremely powerful counterpoint to understanding brain and behaviour?

    ReplyDelete
  2. The answer to part of this is the previous rant: I don't actually think fMRI can give us even the types of answers you suggest, because I don't think in general the question is posed correctly.

    Second, I think ecological types get a bad rap on the brain. The reason we don't care about the brain is not that we don't think it's interesting; it's that we don't think that studying what the brain does makes sense without realising perception begins in the world, not at the eye. This makes neuroscience hard, actually, and beyond most current methods (you can't move in an fMRI, for example).

    Third, I do take the point that if you could ask the right question (as John does) you might then get an answer of interest to someone interested in what the brain does. But you still end up with the claim that a certain piece of the brain does a certain thing, when actually what happens is that most of the brain does certain things at certain times under certain conditions - there's no way to get back to the system level that, I think, is actually going to be responsible for doing whatever it is that the brain is doing.

    Let me put it this way: even John started his talk by acknowledging that in this task, the entire brain goes off. Say Parts A & B go off in Task 1, while A, B and C go off in Task 2. An fMRI study would conclude that C does whatever is different in Task 2 as compared to Task 1; but what if B actually does different things when working with A than with A & C? I think this is highly likely to be the kind of thing that goes on; the brain is incredibly dynamic and changing at the axon level on millisecond timescales. B isn't a circuit that has been formed by experience; B is a collection of neurons that get recruited into various systems at various times.

    I just think that fMRI is the wrong question. This post was just me thinking that even when done properly, according to the rules and with one eye on the information, you don't learn much.

    Keep defending imaging though; I might not actually be right (although I obviously don't really think that's true :)

    ReplyDelete
  3. WOW! Andrew actually said "I might not actually be right" -- that's about as close as he ever gets to admitting that he was wrong. :)

    ReplyDelete
  4. You'll notice I followed that up right away with a qualifier :)

    ReplyDelete
  5. But does John ask the right question? I've not actually read the paper, but given the title, I assume it is determining the neural correlates of steering yourself (or seeing yourself steered about). Determining these neuronal mechanisms for the sake of determining these neuronal mechanisms seems to be the sort of (very expensive) who cares study you were unhappy with, irrespective of how well controlled the stimuli were.

    ReplyDelete
  6. By 'ask the right question' I mean he presented the nervous system with ecologically valid information that he had complete control over. He didn't just have a video of some driving.

    And I am indeed still saying 'who cares' about this. I found that even when done right it still wasn't satisfying and that was interesting.

    ReplyDelete