Why Most Published Research Findings Are False

Show general stats for sri.

God
Administrator
Full Member
Karma: 65535
Posts: 132

Why Most Published Research Findings Are False

Quote

"Researchers have found that the winner's curse may apply to the publication of scientific papers and that incorrect findings are more likely to end up in print than correct findings. Dr John Ioannidis bases his argument about incorrect research partly on a study of 49 papers on the effectiveness of medical interventions published in leading journals that had been cited by more than 1,000 other scientists, and his finding that, within only a few years, almost a third of the papers had been refuted by other studies. Ioannidis argues that scientific research is so difficult — the sample sizes must be big and the analysis rigorous — that most research may end up being wrong, and the 'hotter' the field, the greater the competition is, and the more likely that published research in top journals could be wrong. Another study earlier this year found that among the studies submitted to the FDA about the effectiveness of antidepressants, almost all of those with positive results were published, whereas very few of those with negative results saw print, although negative results are potentially just as informative as positive (if less exciting)."

http://science.slashdot.org/article.pl?sid=08/10/19/172254

http://www.grok.in/
"Ignorance killed the cat, curiosity was framed."

October 21, 2008, 09:21:52 am

#1

sri

sri

Show sri's last posts.

Administrator
Sr. Member
Karma: 7
Posts: 264

Re: Why Most Published Research Findings Are False

Looks like this is more about clinical research where the effectiveness of drugs and practices are statistically established. Very unlike for example, a theoretical paper where results are proved by theorems.

Quote

Dr Ioannidis based his earlier argument about incorrect research partly on a study of 49 papers in leading journals that had been cited by more than 1,000 other scientists.

Besides how good is a sample of just 49 papers? Also among the "more than 1000 citations", what were the nature of these citations? Were self-citations removed? Was it established that the citations were non-nepotistic (by a third, independent party without any social/professional links to the authors)? Were the citations meant to endorse the results or challenge them in the first place?

Looks like the results of this paper are also as believable as the papers whose results it seeks to trash..

October 21, 2008, 10:24:56 am

#2

sids

sids

Quote from: sri on October 21, 2008, 09:21:52 am

God
Administrator
Full Member
Karma: 65535
Posts: 132

Re: Why Most Published Research Findings Are False

Looks like the results of this paper are also as believable as the papers whose results it seeks to trash..

Ironic, isn't it?

I agree with most of your observations. I mostly posted this because the articles to which this post links to are pretty interesting. So is the reference to 'The Winnder's Curse'.

For easier reference, here are the articles the post links to:

http://www.economist.com/science/displaystory.cfm?story_id=12376658

Quote

One group of researchers thinks headline-grabbing scientific reports are the most likely to turn out to be wrong.

IN ECONOMIC theory the winner’s curse refers to the idea that someone who places the winning bid in an auction may have paid too much. Consider, for example, bids to develop an oil field. Most of the offers are likely to cluster around the true value of the resource, so the highest bidder probably paid too much.

The same thing may be happening in scientific publishing, according to a new analysis. With so many scientific papers chasing so few pages in the most prestigious journals, the winners could be the ones most likely to oversell themselves—to trumpet dramatic or important results that later turn out to be false. This would produce a distorted picture of scientific knowledge, with less dramatic (but more accurate) results either relegated to obscure journals or left unpublished.

http://medicine.plosjournals.org/perlserv/?request=get-document&doi=10.1371/journal.pmed.0050201&ct=1

Quote

Why Current Publication Practices May Distort Science

Summary: The current system of publication in biomedical research provides a distorted view of the reality of scientific data that are generated in the laboratory and clinic. This system can be studied by applying principles from the field of economics. The “winner's curse,” a more general statement of publication bias, suggests that the small proportion of results chosen for publication are unrepresentative of scientists' repeated samplings of the real world. The self-correcting mechanism in science is retarded by the extreme imbalance between the abundance of supply (the output of basic science laboratories and clinical investigations) and the increasingly limited venues for publication (journals with sufficiently high impact). This system would be expected intrinsically to lead to the misallocation of resources. The scarcity of available outlets is artificial, based on the costs of printing in an electronic age and a belief that selectivity is equivalent to quality. Science is subject to great uncertainty: we cannot be confident now which efforts will ultimately yield worthwhile achievements. However, the current system abdicates to a small number of intermediates an authoritative prescience to anticipate a highly unpredictable future. In considering society's expectations and our own goals as scientists, we believe that there is a moral imperative to reconsider how scientific data are judged and disseminated.

I think some of the arguments they put forth are very good although the conclusion (especially in the former article) is a little too specific. It is probably sufficient to say that the research published in the best scientific journals _may_ not really be the best research that is happening. This is a very difficult conclusion to reach anyway because of it's self-contradictory nature ('best' is presently defined by the endorsement by the journals).

http://www.grok.in/
"Ignorance killed the cat, curiosity was framed."

October 21, 2008, 10:27:59 am

#3

sri

sri

Show sri's last posts.

Show general stats for sri.

Administrator
Sr. Member
Karma: 7
Posts: 264

Re: Why Most Published Research Findings Are False

Agree with the general "winner's curse" phenomena being played out in scientific publishing though.. Except that the methodology taken up by the said paper wasn't all that convincing.

October 21, 2008, 05:57:55 pm

#4

sanket

sanket

Show sanket's last posts.

Show general stats for sanket.

Global Moderator
Full Member
Karma: 4
Posts: 104

Re: Why Most Published Research Findings Are False

I've not gone through the cited articles; this is more of a general comment about published work and the argument that "most published research findings are false". What we should realsise is that the problems researchers, in any domain, address today are either too complex to make a groundbreaking impact in one shot, or are reaching a level of maturity where such groundbreaking new results are hard to come by. So, I'd like to rephrase the above statement as - "Why Most Published Research Findings are only a delta-x improvement over the state-of-the-art, that too under several realistic or unrealistic constraints?" The PhD comic that says - "There's this problem, right? That's really important... and all the research that's been done before doesn't quite solve it. But now *your* solution kind of solves it ... But only under certain conditions and assumptions that may or may not apply to reality," is not untrue. (http://www.phdcomics.com/comics/archive.php?comicid=864)

It is simply inevitable that most published work is delta-x contribution. I was once listening to an old lecture by Donald Knuth in which he makes an argument similar to this. He asks, if you check the journals or conferences in computer science every year, how many major results do you find? May be one or two, or even none. However, let's say you are sleeping for 5 years and wake up suddenly to look what is new, you'll find that the field has changed dramatically! It is these numerous tiny contributions that come together to make a total impact, rather than a small set of major results making a huge impact.

So, I feel it is harsh to say that most results that are published are false, unless there is deliberate fraud or even willful negligence, even if the results are refuted in due course. As I said, the problems are too complex to pick out on results in isolation and judging them to be "false" or "true". In the field of medicine it is especially hard to establish exact results.

October 21, 2008, 06:36:23 pm

#5

sids

sids

Show sanket's last posts.

God
Administrator
Full Member
Karma: 65535
Posts: 132

Re: Why Most Published Research Findings Are False

Well said, sanket. I urge you to read the links if you can spare the time, their argument is on similar lines too. They do note that what they are referring to is not malice in any way, just that the process has problems. In fact, the later (which is actually a paper) does not make any such accusations. The former does so primarily due to perversion by the journalist (I find it quite ironical).

http://www.grok.in/
"Ignorance killed the cat, curiosity was framed."

October 21, 2008, 08:14:31 pm

#6

sanket

sanket

Show general stats for sanket.

Global Moderator
Full Member
Karma: 4
Posts: 104

Re: Why Most Published Research Findings Are False

@ sids: Yes, I read the articles now, and to be fair, the PLOS paper does raise important questions about current publishing practices. Chances are there that many of the things such as winner's curse, oligopoly, artificial scarcity, might be happening. And it's very true that most of the articles published in top journals are not very accessible. So, those are important questions and might well be causing hindrance in the growth and dissemination of science.

However, it is difficult to get convinced that these are the factors that are dominating most of the scientific research that's going on. The idea that what we see as a rigorous peer-review process is __mostly__ eyewash or gimmick, is, well, terrifying, if not anything else. And can we not extrapolate this argument? When the most prestigious organizations recruit people for top positions, do they mostly recruit less merited people than they could have?

The Economist article, IMO, is more clever than necessary. "The question for Dr Ioannidis is that now his latest work has been accepted by a journal, is that reason to doubt it?" what about that?

October 22, 2008, 07:15:14 am

#7

sids

sids