The blog title is taken from The Long Dark Teatime of the Soul, by Douglas Adams. If you haven't read it, I urge you to give it a go.
Anyway, that's largely irrelevant. Shortly after choosing the title for this blog, my father sent me an article (this one) which mentions a paper entitled "Why Most Published Research Findings Are False". I decided that issue was a good starting point!
Last year I got pretty low thinking about the many ways psychological research findings can be false. We think of science as objective, but in reality it's often far from that.
In the first place, most findings are based on statistical tests, which are officially 'significant' if they would occur by chance in less than 5% of cases. Of course, if you do 20 tests, 1 would be significant by chance. And many research studies include a lot more comparisons than that! We could set the 'significance' level higher, but of course that doesn't mean that the results aren't due to chance.
There's the issue of publication bias - it's well known that statistically significant results are more likely to get published. Less obvious is that results that are surprising, but plausible, are the most likely to get published. So new findings that seem to be elegantly explained by a new theory will get published, but once that theory has been around for a while, findings that disprove that theory are more likely to get published. So research goes in waves, 'proving' and 'disproving' ideas in turn.
Finally, studying psychology has it's own issues. Most cognitive terms are in some sense a metaphor, and of course all metaphors are wrong in some ways. For example, I am interested in phonological representations - in the representation of the sound structure of words. After studying them for over 10 years, I have no idea what form these 'representations' take. In reality, no one does. Even if we could pull apart the brain and see the patterns of activation that match up to each word, it wouldn't really answer the questions of how these representations work. There would still be an explanatory gap.
I wanted to make three quick points about this topic.
ReplyDeleteFirst, which we all know, it highlights the dangers of “mining the data”. A probabilistic model of confirmation/rejection means something very different in relation to a previously-stated hypothesis from what it means in relation to a large survey data set, for example. The kind of sample-based national assessment I do regularly may well have 100 or more variables, and few or no actual hypotheses. In that, it is a classic case of mining the data. A statistically significant finding from such a survey can be reported, but should only be taken seriously when it is replicated, preferably in a designed experiment.
The second thing is that even with respect to a designed experiment with a stated hypothesis, one finding is hard to interpret in isolation. What really matters is the distribution of findings, rather than the individual results. That, after all, is what replication is about. Therefore, we need to know how many studies have been undertaken on a given topic, and have access to the outcomes of all of them, before we can judge an individual result.
The third thing, which I had not thought of until breakfast time this morning, but which follows logically, is that we should be extremely cautious about how we interpret meta-analysis. Superficially, meta-analysis seems like a great idea, but if the universe of available findings is systematically skewed in the way that is suggested, this surely casts severe doubt on the value of meta-analysis?
So it seems to me that what is really needed is much better regulation – not in the sense of restricting access, but making sure that we know how many related results were inconclusive or negative, and so can put “significant” findings in a better context. I see two immediate problems with that. First, it implies a move away from the current system of peer-reviewed, exclusive and expensive academic journals to some kind of centralised, open-access (but still peer-reviewed) registry, with ongoing real-time meta-analysis. Second, the vested interests – commercial funders, publishers – will never let that happen.
Hmm. I guess that it what the guidelines for RCT and other medical trials aim to deal with - you have to register the hypotheses and planned analyses before the data is collected, so that people can follow up and find out what your results were - you can't bury it! That doesn't really work for more open-ended projects though.
ReplyDeleteRe. meta analyses, best practice is to ask likely people to provide 'failed' or unpublished research studies as well as published ones, but of course there's no way of doing that systematically...