Saturday 7 May 2011

Failing to Replicate Bem's Ability to Get Published in a Major Journal

I think Daryl Bem has done psychology an enormous favour. Possibly even two.

As you probably  know, Bem is the author of 'Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect', a paper with claims to have found evidence for precognition by running standard psychological experiments in reverse and demonstrating small but statistically reliable effects on behaviour of stimuli which came after the response was made. I posted briefly about it here, and otherwise it's been all over all of the internet for months.

It's back on my radar because several psychologists, including Richard Wiseman, recently submitted a failure to replicate the studies to the Journal of Personality & Social Psychology (JPSP), which is where Bem published his work. As reported here, Eliot Smith, the editor, refused to even send this (and another, successful replication as well) out for review. The reason Smith gives is that JPSP is not in the business of publishing mere replications - it prioritises novel results, and he suggests the authors take their work to other (presumably lesser) journals. This is nothing new - flagship journals like JPSP all have policies in place like this. But it's not a good look, and it got me thinking.

The Low Bar

Bem's first favour is, I think, to have exposed the extraordinarily low bar for publication in psychology. With no coherent central theory, JPSP was able to justify publishing Bem's paper because he followed the rules:
  1. The experiments were appropriate to the question. 
  2. The experimental tasks he adapted were all standard reference points in the literature
  3. The analyses were all standard - Bem ran the correct General Linear Model statistics on his reaction time data
  4. The paper was meticulous in it's detail - the studies are fully replicable and reported in clear, straight-forward language.
In fact, I think JPSP almost had no choice but to publish the paper, because the only reason it deserved to fail peer review was because it was a ridiculous question. But the reasons that make it ridiculous (the literal, physical impossibility of what Bem was proposing) live outside the general bounds of psychology and we simply aren't sufficiently well connected to either physics, chemistry or biology in our theorising for that information to be allowed into the decision making process. As evidence, let me quote the editorial which accompanied Bem's article:
We openly admit that the reported findings conflict with our own beliefs about causality and that we find them extremely puzzling. Yet, as editors we were guided by the conviction that this paper—as strange as the findings may be—should be evaluated just as any other manuscript on the basis of rigorous peer review. Our obligation as journal editors is not to endorse particular hypotheses but to advance and stimulate science through a rigorous review process. [italics added]
Think about that: the data conflict with what the editors of a psychology journal believe about causality. They couldn't bring themselves to refer to all the empirical data physics has about causality. They couldn't even rule the paper out based on Bem's handwaving towards quantum mechanics in his Discussion section, which is almost certainly incoherent gibberish that appeals to vague parallels between psi claims and quantum weirdness and isn't even close to a plausible mechanism. JPSP didn't require Bem to provide a mechanism for his outrageous claim, but they were still able to publish the paper.

This is, of course, a ludicrous place for psychology to find itself. Psychology needs to align it's explanations with the natural sciences (at least with biology, if not physics) and in the meantime, we need to stop thinking we know what we're talking about when we witter about quantum mechanics.

Replication
Replication is key in science. Findings become robust and reliable only once they have survived various attempts to break them. Bem's second favour is to expose the well known secret that major journals simply won't publish replications. This is a real problem: in this age of Research Excellence Frameworks and other assessments, the pressure is on people to publish in high impact journals. Careful replication of controversial results is therefore good science but bad research strategy under these pressures, so these replications are unlikely to ever get run. Even when they do get run, they don't get published, further reducing the incentive to run these studies next time. The field is left with a series of "exciting" results dangling in mid-air, connected only to other studies run in the same lab.

JPSP they think they've done their bit to be 'fair and balanced' by publishing the Wagenmakers et al (2011) commentary on the problems with GLM analyses with Bem's paper. But think about the message JPSP is sending to authors. That message is 'we will publish your crazy story if it's new, but not your sensible story if it's merely a replication'. A journal that couldn't find a reason to bounce Bem on sight has no right bouncing better science on this basis.

So thanks, Professor Bem. Your ridiculous research and the Type I errors you managed to find have, at least, gotten everyone talking about some very real problems in scientific publishing, and you've opened a discussion for psychologists about how we should be analysing our data. Bet you didn't see that coming.

References
Bem, D. (2011). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology, 100 (3), 407-425 DOI: 10.1037/a0021524

Wagenmakers, EJ, Wetzels, R,  Borsboom, D,  & van der Maas, HL (2011). Why psychologists must change the way they analyze their data:  The case of psi:  Comment on  Bem (2011). Journal of Personality and Social Psychology, 100(3), 426-432 DOI: 10.1037/a0022790

37 comments:

  1. In all fairness, I think at least some of Bem's results are plausible. The idea that college students can predict where, on a computer, pornographic images are about to appear, is not as crazy as you make it sound sounds.

    In all seriousness (note: that last part was at least partially serious), I think you are right on. While I understand the reluctance to publish "mere" replications, replicate-and-extend and failure-to-replicate should be much more common. This arrogance of the major journals is especially disturbing in light of the increasing respect for meta-analysis. Meta-analysis makes the most sense under the assumption that the experiments being combined are similar in their essential elements. If all the studies are different, then the endeavor (and with it much of the notion of "cumulative knowledge") makes little sense.

    P.S. Paul Meehl wrote some killer papers about this problem. His paper about why literature reviews are uninterpretable gives you that feeling where you want to laugh and cry at the same time.

    ReplyDelete
  2. You know that Bem's paper is just the first to be published in a mainstream psychology journal right?

    And that the effect has been studied since at least 1997?
    http://scholar.google.com/scholar?q=radin%2C+dean&hl=en&btnG=Search&as_sdt=1%2C5&as_sdtp=on

    Its also been replicated by a number of different authors, mostly published in the journal of scientific exploration?
    http://scholar.google.com/scholar?hl=en&q=presentiment&btnG=Search&as_sdt=0%2C5&as_ylo=&as_vis=0

    I'm not in any way endorsing the findings and I agree that psychological publishing has many problems, but this effect (apart from its strangeness) appears to be in need of an explanation.

    ReplyDelete
  3. "So thanks, Professor Bem. Your ridiculous research and the Type I errors you managed to find have, at least, gotten everyone talking about some very real problems in scientific publishing, and you've opened a discussion for psychologists about how we should be analysing our data. Bet you didn't see that coming."

    ...Excuse me but are you 12 years old?

    ReplyDelete
  4. No, just unimpressed with research into a phenomenon that can't exist. One of the perks of being a scientist is that I don't have to put up with rubbish like this when I have good reason to call it out.

    ReplyDelete
  5. seriously...you think that your "rubbish" pov makes you a champion of "True Science"? It doesn't...it makes you more dangerous to science than a thousand Bem's.

    The peer review here is..." re: Wilson's paper...biased, childish and self agrandizing. Dangerous acceptance of assumptions not substantiated in the work...not worth reading or studying regardless of methodology as it is clear that data would be skewed from moment one."

    Wheeeee! Welcome to your brave new world of science via inquisition. Hope you enjoy the monster you are helping to create.

    ReplyDelete
  6. On the one hand, you say

    "we need to stop thinking we know what we're talking about when we witter about quantum mechanics"

    and on the other, you say

    "Bem's handwaving towards quantum mechanics in his Discussion section, which is almost certainly incoherent gibberish".

    You can't have it both ways. If you don't know what you're talking about when it comes to quantum mechanics then there is no way you could judge how likely it is that Bem's discussion of quantum mechanics is incoherent gibberish.

    This post is just absurd, full of childish statements and illogical arguments.

    ReplyDelete
  7. You can't have it both ways. If you don't know what you're talking about when it comes to quantum mechanics then there is no way you could judge how likely it is that Bem's discussion of quantum mechanics is incoherent gibberish.
    This isn't having it both ways; these two points are the same point. The reason I can identify there is a problem with Bem's use of quantum theory is that there is no maths about quantum physics in his paper, just words. About the only thing I do know about quantum physics is that it really only makes sense mathematically.

    And I don't really agree that I'm being childish here - I stand by what I say about this issue, and I really do actually worry about these issues in my discipline.

    And so:
    seriously...you think that your "rubbish" pov makes you a champion of "True Science"? It doesn't...it makes you more dangerous to science than a thousand Bem's.
    I think 'rubbish' is just an accurate summary of my feelings about this issue, feelings which I actually spend some time fleshing out in detail here. I've laid out quite a few reasons for my views, don't get distracted by one word.

    ReplyDelete
  8. You're doing it again. Why is it that Bem's knowledge of quantum physics is less valid that the your (single item) knowledge of quantum physics?

    I find it suspiciously convenient that the one thing you think you know about quantum physics just happens to undermine the person you're criticising.

    I think you don't know anything at all about quantum physics and should follow your own advice: say nothing. Leave it to the physicists to comment on the parts of Bem's paper that deal with physics. Childish remarks about something you know nothing about only serve to make you appear absurd, childish and illogical.

    ReplyDelete
  9. Part of analysing a scientific claim is analysing the form of the argument it is embedded in. The form of Bem's appeal to quantum theory has many features which make me suspicious, and which don't require any knowledge of the details of quantum mechanics to critique. I would only need to be a physicist to critique an attempt to formally model these results, because then we'd be in territory that required the specialist knowledge. Bem isn't even close to that territory.

    To be a little more specific:

    First, the argument is to merely note parallels between the verbal descriptions of quantum phenomena and aspects of psi (pg 16). Bem attempts to head off criticism of this by pointing out that not everyone thinks quantum will explain psi and that even if it does it still won't be very intuitive because quantum is just weird. But this simply highlights how weak and ill-defined the appeal to quantum theory is.

    Second, the critical problem with this argument is that there is no attempt to establish a mechanistic link from these atomic and sub-atomic level phenomena to the hot, wet system we call the brain. This needs to be done. There was some recent work showing that photosynthesis takes advantage of quantum phenomena to be efficient enough to work; this work was well received because they actually did the required work to actually establish that the effect was genuinely quantum in nature. They didn't merely wave their hands and say 'well, the way we describe it sounds a bit quantum', which is all Bem and his fellow psi researchers have ever done.

    This work shows that a) quantum phenomena can show up in hot systems but b) still at the molecular level, not the systems level of whatever the hell consciousness is (required for Bem's results), and also c) that if you want to make these kinds of claims, there's a lot of hard science to be done to establish that your proposed mechanism is viable. Bem does none of this; the net result is that his appeal to quantum theory contributes nothing to his argument.

    Finally, he points the way to this trouble himself - his argument is 'our description sounds like it could be a bit like that', but he then cites the famous Feynman quote that ends 'Nobody knows how it can be like that'. Quantum theory makes perfect sense in it's mathematical form, but, as Feynman knew, it makes no obvious sense in the verbal description, and it's that level that Bem is using.

    So my answer to your critique is that the form of Bem's appeal to quantum physics is readily seen to be pointless because he never does any of the actual work required to establish a quantum mechanism. Being able to see this sort of problem with the form of an argument is one of things you learn doing science, because it's a critical part of the game, just as important as the data are.

    So:
    Childish remarks about something you know nothing about only serve to make you appear absurd, childish and illogical.
    I don't need to know anything about quantum physics, just what it takes to make a compelling scientific argument. This is something I do, actually, know quite a lot about, so I'm not doing what you think I am doing.

    ReplyDelete
  10. You've done it again. Your comment is chock full of references to quantum physics, couched in arguments that clearly demonstrate a belief that you know what you're talking about when it comes to quantum physics. You're a hypocrite. You're not doing what you've told others to do. But let's put that aside for the moment because I can see that you're not going to take on board your own comments.

    I'm not sure what your point is. In a paper, Bem discusses some parallels between the concepts used in physics and the concepts in and around the subject of the paper. He's explicitly noted that not everybody believes the parallels are valid, and even that the concepts in physics are not fully understood in the first place.

    What is it you believe he's done that's bad?

    ReplyDelete
  11. He's tried to explain his physically impossible effects using physics he doesn't especially understand (as evidenced by his lack of formal treatment of the physics) and that probably doesn't apply to the system he's interested in (the human brain, rather than something at the atomic level). That, to me, is just some really bad science, and I'm annoyed that he tried it and disappointed that he got away with it). That's about it, in a nutshell.

    And on the previous point: I'm allowed to have sentences that contain the words 'quantum physics' without being a quantum physicist. That's essentially all I've done.

    ReplyDelete
  12. Having reviewed the discussion section of Bem's paper, I can't find any attempt to explain the effects that are observed, only attempts to highlight possible areas where theories may emerge in future. Let me quote:

    "Many psi researchers see sufficiently compelling parallels between these [quantum] phenomena and characteristics of psi to warrant considering them as potential candidates for theories of psi." (p.422)

    "even if quantum-based theories eventually mature from metaphor to genuine models of psi" (p.423)

    These statements clearly show only a qualified hat-tip to possible theories rather than an attempt to provide a full explanation of psi effects using quantum theory. You have mischaracterised Bem's discussion.

    Bem has not tried to explain the data he has, nor would one expect an explanation because at present, there is no such thing.


    You've used the word "impossible" and yet you claim to be a scientist. You do not seem willing to admit that there are boundaries to the knowledge of science and to physics. Physical theories are only models of the behaviour of the universe, they are not complete descriptions. We have no Grand Unified Theory.

    All theories are based on assumptions and it is only when those assumptions are true and one is working within a model that the word "impossible" makes sense. One cannot prove that a model's assumptions are correct from within the model itself.

    We cannot say that anything is impossible because we do not have a complete understanding of the universe. You claim to be a scientist so you should know this.

    ReplyDelete
  13. People only hat-tip to quantum physics like this when they want to say 'sure my claims make no sense; but hey, neither does quantum physics and they're a real science!'

    Roger Penrose did this in 'The Emperor's New Mind' - his argument went

    1. Consciousness is mysterious.
    2. Quantum physics is mysterious.
    3. By Occam's razor,we can simplify our science by assuming that 1 and 2 are the same mystery. BAM. Consciousness solved.

    Bem is doing the same thing.

    It's nice that you are thinking about the limits of our knowledge. But right now, psi is considered as impossible based on what we know about a lot of things; so a healthy scientific response is

    a) given psi is impossible, you can't use it in your explanation for your data. This means you need something else.
    b) First swing: problems with data analysis. See Wagenmakers et al, 2011. (This actually identifies the problem, so you can stop here in this case).
    c) If that hadn't worked, you could try to replicate the results. See Wiseman et al, who have failed to replicate, if they ever find a journal willing to accept a mere replication (see this post).

    Only if the analysis had held up and the results had been replicated would I be under any scientific obligation to start taking this seriously. In the meantime, I start from the point of view that psi is impossible given our current understanding of how things work, and am reassured about this by the problems and failures to replicate.

    Scientists rule potential mechanisms out as impossible every day - you can't do science if you think everything is always on the table and we often have a lot of data about why something won't work. Sometimes we're wrong, but it takes hard work (of which Bem does very little) to show this, and science has mechanisms which allow these errors to be identified.

    ReplyDelete
  14. "People only hat-tip to quantum physics like this when they want to say 'sure my claims make no sense; but hey, neither does quantum physics and they're a real science!'"

    If you think about it, this is ridiculous. Nobody would say that their claims make no sense. What this is, in fact, is you projecting your lack of understanding onto others. It's a statement by you, that the data does not make sense to you. That's not a bad thing; nobody fully understands the precognition data yet.


    You've claimed that Bem is following Penrose's argument, which you've simplified. Assuming for the sake of argument that your simplification is correct, it has no relevence to Bem's paper. You've claimed that Penrose considers consciousness to be solved. Bem does not consider precognition to be solved.

    Bem does not draw parallels between such high-level generalities as 'mysteriousness'. Instead, he draws parallels between highly specific properties of both, in particular nonlocality and the validity of retrocausation.

    Bem is not doing the same thing as Penrose.


    "right now, psi is considered as impossible based on what we know about a lot of things"

    This is just vague hand-waving. There is nothing that could cause psi to be considered impossible. Because we do not have a complete understanding of the universe, we cannot claim that anything is impossible. What do you believe we know that makes psi impossible?


    "a) given psi is impossible, you can't use it in your explanation for your data"

    This statement doesn't make sense and shows a clear lack of understanding about what the word "psi" means. The word "psi" does not refer to any specific process of phenomena. Instead, it is essentially a place-holder or wild card. It refers to 'whatever process gives rise to the data'. (Note that this could equally refer to a normal process as a paranormal process.)

    Replacing this definition in the sentence above, we get "given whatever process that gives rise to the data is impossible, you can't use it in your explanation for your data". This is a nonsense sentence, it doesn't make sense. It is essentially saying 'the data are not possible so you cannot use them in your explanation for the data'.

    Firstly, it doesn't make sense to try to use the data in an "explanation" for the data itself; you're essentially describing a tautology (which is meaningless). Secondly, you're claiming that the data are not possible and yet you're referring *to* the data. If the data are not possible then there would be nothing to refer to.

    Here I am not addressing the validity or possibility of the data or any supposed explanations. Instead, I am simply pointing out that your use of the word "psi" shows you do not understand what it means and that what you've said about psi is nonsense; gibberish.


    "b) First swing: problems with data analysis. See Wagenmakers et al, 2011. (This actually identifies the problem ..."

    If we're going to see Wagenmakers et al. (2011) then let's also see Bem, Utts and Johnson (2011) http://dbem.ws/ResponsetoWagenmakers.pdf where Wagenmakers' analysis is shown to use inappropriate mathematical terms in critical places. And further, let's see again Wagenmakers et al (2011b) http://www.ruudwetzels.com/articles/ClarificationsForBemUttsJohnson.pdf where Wagenmakers et al. retreat from their original criticisms and instead rely solely on tenuous general modelling to imply only a possible problem.

    ReplyDelete
  15. But why stop there, let's look at Alcock (2011) http://www.csicop.org/specialarticles/show/back_from_the_future where Alcock tries to identify problems in Bem's article and claims that we should dismiss the entire work because, using an analogy from chemisty, Bem's test tubes are dirty. Then let's look at Bem (2011b) http://www.csicop.org/specialarticles/show/response_to_alcocks_back_from_the_future_comments_on_bem where Bem shows that Alcock is missing a basic grasp of the mathematics used and that his arguments are nonsense. Then look at Alcock (2011b) http://www.csicop.org/specialarticles/show/response_to_bems_comments and see a surly, hypocrticial response where Alcock refuses to acknowledge that his own test tubes are dirty.

    Let's go further and look at Wiseman (2011) http:/richardwiseman.wordpress.com/2010/11/18/bems-esp-research/ who points out one of the few valid criticisms of Bem's paper, a methodological flaw. But also see the end of that article where Bem responds and shows that, when taking this flaw into account in the data analysis, the results are changed only negligibly and the statistical significance is not effected.

    Of couse, this is only the literature around Bem's article. Really, we should be referring to the wealth of literature on precognition prior to Bem's recent paper. For example, Honorton and Ferrari (1989) http://www.lfr.org/LFR/csl/library/HonortonFerrari.pdf examined 309 studies from 1935 to 1987 in a meta-analysis and found a significant overall precognitive effect.

    So you can see that Wagenmakers did not identify "the problem". In fact, Bem's work has stood up to criticism with notable robustness. Considered within in the context of the general precognition literature, the only "problem" here seems to be the ignorance and causal misinterpretation of the literature on your part.


    "so you can stop here in this case"

    This statement is very telling and shows a big problem. Prior to this, you've used the words "a healthy scientific response". In fact, this idea of stopping is extremely unhealthy. What you're recommending people do, and what it seems you've done, is to stop reading the scientific literature as soon as you find a conclusion that is line with your expectations.

    This is extremely unscientific. It is starting with a conclusion, finding evidence for it and not bothering to look at the evidence against it. There is another group of people who work in this way: creationists. Your behaviour in evaluating the precognition literature appears the same as creationists' behaviour in evaluating planetary science.


    "c) If that hadn't worked, you could try to replicate the results. See Wiseman et al, who have failed to replicate"

    Well, I won't see just Wiseman et al; I think I'll also look at Batthyany (2010) http://ssrn.com/abstract=1715954 who attempted to replicate a subset of the studies in Bem's paper, and succeeded.

    Again, it seems you've stopped looking at the literature as soon as you found something in line with your explanations.


    "I start from the point of view that psi is impossible ... and am reassured about this by the problems and failures to replicate."

    Again, this is a telling sentence. Look at the key words and phrases you use: "impossible", "stop", "reassured ... by the ... failures".

    These are not the words of a person who is open-minded and evaluating the data honestly and with integrity; ie, a scientist. Instead, they are the words of a person who is afraid by the idea that has been put before him and is grasping on to anything, no matter how tenuous, that might allow him to push the idea away.

    Why are you starting from the point of view that psi is impossible?

    ReplyDelete
  16. "Scientists rule potential mechanisms out as impossible every day - you can't do science if you think everything is always on the table and we often have a lot of data about why something won't work."

    Bem's paper is not about a mechanism; it is simply the presentation of some data. The mechanism is not understood. Specific mechanisms that have been put forward as possible candidates for psi will be ruled our or not according to particular models or, if they do not rely on particular models, specific data or, they will not be ruled out.

    Regardless, Bem has put forward no such mechanism.


    "Sometimes we're wrong, but it takes hard work (of which Bem does very little) to show this"

    I find it extraordinarily hypocritical that you, who it seems simply hasn't bothered to look at the literature, would criticise Bem for doing too little work. Your criticism of Bem shows too little work.

    ReplyDelete
  17. I'll get to the rest when I have a little time to do it properly, but I have an answer for this:
    What do you believe we know that makes psi impossible?
    The impossibility of time travel

    ReplyDelete
  18. OK, some comments now I've had 5 minutes to think this morning.

    This statement doesn't make sense and shows a clear lack of understanding about what the word "psi" means. The word "psi" does not refer to any specific process of phenomena. Instead, it is essentially a place-holder or wild card. It refers to 'whatever process gives rise to the data'.
    Incorrect. Psi, for this paper, is the hypothesis that information can travel backwards in time in a format the brain can register and use at some level. It is not as generic as you say, so the rest of your analysis doesn't hold.

    In addition, there is no such things as theory-free observation. This is the first thing that comes up in any philosophy of science course. In science, there are the data, and there is your proposed mechanism for the production of that data. But you cannot merely 'report the data' because they are meaningless without the context of the theory which motivated that experimental design, etc. This is also, interestingly, part of the Bayeisan critique of the GLM model of analysis, I think.

    Either Bem is proposing a quantum mechanism, in which case I think he has revealed himself to have no clear story, or he is not proposing a mechanism, in which case his hat tipping to quantum physics is mere noise ('look - their stuff is weird too!') and his data can't be interpreted.

    It's true I haven't dedicated my life to this topic; I have many more interesting things to do. But the reason I haven't is that Bem's work fails to pass a basic scientific 'sniff test'. I;m not afraid of his work - I am entirely unconvinced by it. Your rather literal minded and superficial analysis doesn't really alter that fact.

    ReplyDelete
  19. You've linked to a popular magazine article that claims time travel is impossible because a physicist has shown that a single photon does not exceed the speed of light.

    It isn't clear why you believe the light barrier precludes psi. You seem to be arguing against a theory of psi that depends on something travelling faster than light. As far as I know, there is no such proposed theory from serious parapsychologists (like Bem).

    Your argument is a straw man.

    However, I asked you what made you believe that psi was impossible and you've answered that question, thank you. Now I ask you a further question in an effort to get to the heart of the matter: What made you believe that psi depended on time travel?


    "Psi, for this paper, is the hypothesis that information can travel backwards in time in a format the brain can register and use at some level. It is not as generic as you say, so the rest of your analysis doesn't hold."

    You are wrong. Let's look at the first sentence of the abstract of Bem's paper (which is identical to the first sentence of the text):

    "The term psi denotes anomalous processes of information or energy transfer that are currently unexplained in terms of known physical or biological mechanisms."

    Here Bem is presenting a generic definition very similar to my own. Now let's look at the second sentence in the abstract:

    "Two variants of psi are precognition (conscious cognitive awareness) and premonition (affective apprehension) of a future event ..."

    Here Bem is explicitly distinguishing precognition from the more general idea of psi.

    Again, you've shown that you do not understand what the word "psi" means or how it is distinct from the word "precognition". Moreover, it seems that either you didn't read Bem's paper, or you didn't understand it.

    Psi is, for this paper and all other serious discussion, as generic as I've said. The rest of my analysis does hold.


    "In addition, there is no such things as theory-free observation."

    I have not argued for theory-free observation. Again, this is a straw man.


    "you cannot merely 'report the data' because they are meaningless without the context of the theory which motivated that experimental design"

    You seem to be pointing out tactit knowledge. Yes, you are correct that there are underlying assumptions and theoretical frameworks from which parapsychologists must work in order to produce meaningful data. However, these are not the same as formal scientific theory. They are not stated explicitly in parapsychology nor would one expect them to be for they are not stated explicitly in any field; that is why it described as tacit knowledge and not just knowledge.

    The fact that tacit knowledge exists within the field of parapsychology is not an argument against the data produced by it.


    "Bem is ... not proposing a mechanism, in which case his hat tipping to quantum physics is mere noise ('look - their stuff is weird too!') and his data can't be interpreted."

    You are right, Bem has not proposed a mechanism and his data can not be interpreted. That is why we use the term "psi", a place holder for an unexplained process.

    The fact that the data cannot be explained does not mean it is invalid. Look at, for example the discovery of X-Rays; Roentgen's X-Ray data shocked the world of physics and could not be explained for years but his data was valid. The absence of a formal scientific theory does not mean Bem's data is invalid.

    You keep referring to quantum physics as if Bem's data depends on it. It does not depend on it and Bem has made no claim that it does. His discussion of quantum theory is simply the elucidation of leading theoretical approaches in the field. In scientific papers, such elucidation is common and lauded.

    ReplyDelete
  20. "It's true I haven't dedicated my life to this topic; I have many more interesting things to do."

    You don't have to dedicate your life to a topic in order to get things right.

    ReplyDelete
  21. For information to arrive from the future, it must effectively travel faster than light. For that information to affect judgements made by a person, that information must be in a format usable by the brain. Bem's description is vague, but essentially contains both these premises; given the impossibility of time travel, as understood by modern physics, I'm allowed to start from the premise that psi (both precognition and premonition) is impossible.

    You started on quantum physics. I've just been pointing out why it doesn't help Bem. The issue of theory-dependence of data kicks in here: without the magic of quantum mechanics to provide a veneer of respectability, his claims are ludicrous, and his data meaningless.

    On that subject: in the social sciences we depend on inferential statistics. These come with the risk of Types I and II errors, which we suck up on the understanding that replication will identify when these have occurred. Without a clear and useful mechanism, Bem cannot be sure he hasn't made a Type I error without replication, and the entire point of this particular post was to be grumpy at how those replications, which support the Type I error interpretation, are being blocked (that replication actually came out of someone's file drawer and doesn't count as a replication of Bem).

    So his lack of a valid mechanism is critical, and the existence of informative failures to replicate is important.

    ReplyDelete
  22. "given the impossibility of time travel, as understood by modern physics"

    Here, I think, we get to the heart of the matter. You have reigned in your use of the word "impossible" and qualified it with the words "as understood by modern physics". This is the critical turn where we can see that, in fact psi and precognition are not impossible but simply inconsistent with current physics.


    "I'm allowed to start from the premise that psi (both precognition and premonition) is impossible."

    No, you're not. Impossibility is not a premise. If you say that something is impossible, you have nowhere to go. You cannot "start" there; either it is impossible *or* you start with a premise that can be proved wrong. You cannot "start" with something that might be disproved and call that thing "impossible".

    Again, it is clear that your use of the word "impossible" was an error and what you actually meant was just 'inconsistent with modern physics'.


    "The issue of theory-dependence of data kicks in here: without the magic of quantum mechanics to provide a veneer of respectability, his claims are ludicrous, and his data meaningless."

    You seem to be claiming again that data are meaningless and ludicrous without a theory to back them up. As I noted in my previous comment, that is not the case. The history of the discovery of X-Rays shows that data can arise that appear ludicrous but which have meaning and validity. That you believe Bem's data are ludicrous means nothing; it does not invalidate them or make them any less meaningful.

    You're still clinging desperately to the discussion of quantum physics as well. I don't know why you keep referring to it. Bem hasn't claimed any value due to quantum theory. The value in his data is not dependent on any particular theory, quantum or otherwise. Bem doesn't need "help"; the data speaks for itself.


    "the entire point of this particular post was to be grumpy at how those replications, which support the Type I error interpretation, are being blocked"

    You went a quite far off-course in your post and your comments. Regardless, that's fine, be grumpy about journals. Just don't expect your views to be taken seriously when you use words like "impossible" and spout nonsense.

    ReplyDelete
  23. Imagine that Bem had invoked the ether in his paper. No one would begrudge me pointing out that modern science believes there is no such thing, and therefore it's not an entity you can use in your theorising. Now remember that Bem has invoked time travelling information, and that modern science is currently full of evidence there can be no such thing.

    ReplyDelete
  24. "Imagine that Bem had invoked the ether in his paper. No one would begrudge me pointing out that modern science believes there is no such thing"

    Actually, and this is quite ironic, I think you'll find there are modern ideas of an aether: http://en.wikipedia.org/wiki/Aether_theories#Contemporary_Ideas .

    Regardless, the problem is that you've confused your own personal beliefs (that precognition is impossible) with modern science's beliefs (that precognition is only inconsistent with modern physics).


    "therefore it's not an entity you can use in your theorising."

    This is a strange statement; you're arguing to prohibit all future paradigm changes. You're saying there can be no valid theories that contradict current scientific knowledge, even if there are data to support it. If that were the case, no new theories would ever emerge and science would stagnate.

    There are no ideas which are barred from theorising if the data allow.

    Regardless, Bem has not presented a theory.

    ReplyDelete
  25. You can't rely on such an entity as part of a mechanism for a separate process. You can, of course, conduct science about whether such an entity actually exists. Bem does the former; anyone wanting to rock the paradigmatic boat can do the latter. They are quite distinct and not contradictory, and have nothing to do with my personal preferences at all.

    This is the basis of various flavours of realism, mostly Hacking's entity realism (which came up in Chapter 9 of Chemero's book which I blogged about recently). I recommend you bone up on those, and maybe some work on how scientists actually go about their business (actually, Hacking's work might be useful for that too; I'll see if I can think of other examples).

    ReplyDelete
  26. Regardless, Bem has not presented a theory.
    Also, this is a problem, not a reason he doesn't have to worry about these things.

    ReplyDelete
  27. "I recommend you bone up on ... how scientists actually go about their business"

    This is the height of conceit. You are a fool.


    "Regardless, Bem has not presented a theory.
    Also, this is a problem, not a reason he doesn't have to worry about these things."

    It is not a problem for just Bem, it is a problem for parapsychology and science in general. I don't think worry is necessary though as we have a method for solving the problem.

    ReplyDelete
  28. This is the height of conceit. You are a fool.
    Bwhahahaha.

    ReplyDelete
  29. I don't think the Type I error problem is restricted to Bem's experiments on the paranormal (of course). I'd bet that 50% or more -- that's a made up number I'm using to make a rhetorical point -- of the work that's published in journals like JPSP, Psychological Science, etc. (i.e,. the journals that like sexy or counter-intuitive stuff) won't replicate. I blogged myself just today on one typical example (in my mind) from Psyc Science here:

    http://spotlightmind.blogspot.com/2011/08/do-you-know-study-showed-that.html

    Please pardon the promotion -- I just want to see more of a discussion of this issue.

    ReplyDelete
  30. Hi Neil

    I'm not much of a fan of Psych Science either, for the reasons you cite (eg this rubbish). That study you blogged about was terrible; Sabrina and I came up with about three things they could have done with that sort of experiment that might have been interesting in about 5 minutes, but of course they all required quite a bit of work and more space that Psych Science usually allows :(

    Bigger journals like JPSP and the JEP series tend to be better, because they allow the space for detailed analyses and careful exposition of longer, more careful studies. But replication is still a problem, both by not being done nor being published when it is done.

    ReplyDelete
  31. Although I largely agree with your comments, I don't think it's helpful to continually call the hypothesis and idea "ridiculous". Theory-based research is all well and good, but let's get real, here - a large part of what experimental psychologists do is effect-chasing, and if they're lucky, the theory can get tagged on later.

    I would encourage any hypothesis that was truly falsifiable, because this allows us to ask questions of nature. How do we know that astrology is nonsense? Because we tested it. Do we need a coherent theory before we test it? No. Putting ideas to the test is what science is all about. If we banned all hypotheses on grounds of "apparent ridiculousness" then we'd be slamming the door on some potentially groundbreaking findings.

    Bem's results are interesting. As the post states, this whole business raises many questions about the scientific process. Compare the coverage that faster-than-light neutrinos got with the response Bem's paper gets. Both present an anomaly. What then happens once an anomaly is reported is the true test of a scientific process. Bem's effect won't replicate. Nothing in parapsychology ever replicates. But Bem has every right to test the hypothesis using the scientific methods at his disposal, and nobody should be allowed to dictate to him or anyone else what the "right" or "wrong" hypotheses are.

    ReplyDelete
  32. a large part of what experimental psychologists do is effect-chasing, and if they're lucky, the theory can get tagged on later.
    This is the problem, not a quaint little quirk we can all live with.

    How do we know that astrology is nonsense? Because we tested it. Do we need a coherent theory before we test it? No.
    Yes you do; you need a reason to run some tests and not others. Those reasons are your theory.

    If we banned all hypotheses on grounds of "apparent ridiculousness" then we'd be slamming the door on some potentially groundbreaking findings.
    I had this out with another anonymous commenter above; I'm not saying we rule Bem out as ridiculous because I think he's wrong. I'm saying we rule it out as ridiculous because physics things he's wrong and we have many reasons to think physics might be on to something. Again, that's what theory is for.

    The neutrino study is actually a good comparison. The authors published the result so people knew what they had found, and the physics community has engaged in a serious effort to break the result. Bem publishes his result and no one can publish the replications (see this post for the latest bullshit on this front). One of us is doing it wrong, and it's not physics.

    ReplyDelete
  33. >How do we know that astrology is nonsense? >Because we tested it. Do we need a coherent >theory before we test it? No.
    >Yes you do; you need a reason to run some tests >and not others. Those reasons are your theory.

    This is stretching the meaning of "theory" a bit. Astrologers can't be said to have ever had a "theory". They had some vague proclamations about effects of X on Y, but the proclamations didn't constitute a theory. It was the vague statement regarding causality that was put to the test.

    Bem had similar reasons for doing his study. "Here is a feature of human experience (ESP/precognition). Does the experience match the way the world works, or is it an illusion?". It was the same logic as any test of astrology would employ; X causes Y.. To test it, he set something up that could falsify the first statement (does it match the world?). Yes, it flies in the face of what makes sense from physics and our understanding of causality, and it appears ridiculous on the basis of this, but anomalies are part of what drives knowledge forward.I'd argue that research that adresses anomalies must be part of the overall story (perhaps a small part, but still a part). Most anomalies will turn out to be error, but some might not. If the scientific process works, we should be able to filter out the genuine anomalies from the error. It seems we're not very good at doing that in psychology, but that doesn't mean we should abandon anything looking at potential anomalies. It just means we need to fix the filter.

    The fact that nobody will publish the replication attempts leaves psychology open to charges of pseudoscientific practices (which I think is one unwritten conclusion of the original post).

    I'm of the opinion that Bem's paper should have been submitted to Nature. Not because I think it's genuine, but because it makes such a huge claim (with data to support it) that it should have been subjected to the most stringent review process possible and thrown open to wider scrutiny.

    ReplyDelete
  34. Go back to the neutrino example. None of the physicists think they've broken relativity yet because it's too well supported. They're looking for the methodological and analytical flaws that led to the erroneous result. Only if that rigorous testing fails to explain the result will anyone even begin to think that there might actually be something super-luminal going on.

    Bem found his results and then claimed he had broken relativity. He poked some of his methods to see what was going on but not his analyses (a critical problem; Wagenmakers et al, 2011). The physicists have a solid place to stand (relativity) and are poking the living crap out of that neutrino result from that secure vantage point; it will either work, or it won't, but either way they'll know the answer. Psychology has no such foundation and so had to publish the study. This then gets in the way of 'filtering out the genuine anomalies', which is of course possible with good scientific practice. That's the problem I'm grumpy about.

    ReplyDelete
  35. Here is the problem with this paper, and it is precisely what Andrew Wilson was arguing from the beginning (I think the argument was lost along the way in the back and forth that commenced in the comments): it is not that there is any methodological flaw or statistical flaw per se by the authors, but rather that in psychology, when we find a significant result, we are able to attribute it to any theory that we choose, often one that is made up. As Andrew stated, theories are limiting factors in that they discard findings that do not make theoretical sense, unless those findings become so numerous as to cast doubt on the theory itself (see Thomas Kuhn for a great analysis of this). For example, one of the Mars rover missions did experiments on Martian soil that typically would indicate the presence life forms. These findings were essentially discarded, however, because theoretically, it is very unlikely that Martian life exists on the surface do the effects of the sun. Thus, these results were significant at the .05 level, which is a fact. How to interpret this fact should be guided by theory, however, as opposed to the other way around. Most likely, these results should be interpreted as interesting but nothing more.

    ReplyDelete
  36. it is not that there is any methodological flaw or statistical flaw per se by the authors, but rather that in psychology, when we find a significant result, we are able to attribute it to any theory that we choose, often one that is made up.
    Yes. Thanks, it's nice to see I did actually make my point there somewhere :)

    ReplyDelete
  37. Well, most studies of course cannot be replicated, and I'm not aware of a theory yet that's not essentially made up.

    ReplyDelete