Should scientists handle retractions differently?

Study: Retracted papers needlessly stigmatize and jeopardize solid research in related fields.

Press Contact

Abby Abazorius
Phone: 617-253-2709
MIT News Office

Media Resources

1 images for download

Access Media

Media can only be downloaded from the desktop version of this website.

It is one of the highest-profile cases of scientific fraud in memory: In 2005, South Korean researcher Woo-Suk Hwang and colleagues made international news by claiming that they had produced embryonic stem cells from a cloned human embryo using nuclear transfer. But within a year, the work had been debunked, soon followed by findings of fraud. South Korea put a moratorium on stem-cell research funding. Some scientists abandoned or reduced their work in the field.

But the case is not so simple: By 2007, other stem-cell researchers had found that the debunked research contained a few solid findings amid the false claims. While prior stem-cell findings remained intact, it took time to rebuild support for the field.

Now a study by MIT scholars quantifies the fallout for scientists whose fields suffer high-profile retractions, with a twist: Even valid older research, when intellectually related to a retracted study, loses credibility — especially if the retracted paper involves malfeasance. The fallout from a retraction does not land solely on the scientists who are at fault, but on people in the field more broadly.  

As the new paper contends, “scientific misconduct and mistakes, as signaled to the scientific community through retractions, cause a relative decline in the vitality of neighboring intellectual fields.” This spillover effect, which includes a 6 percent decline in citations relative to similar, unaffected papers, suggests that scientists would benefit by trying to describe the nature of each retraction in more detail.

“A well-functioning, transparent retraction process is actually part and parcel of the scientific system,” says Pierre Azoulay, an economist at the MIT Sloan School of Management, and a co-author of the new study. “We need a system where … journals help the readers spell out the reasons for the retractions, and help the scientific community parse the implications for the forward movement of science.”

Identifying the “stigma story”

The paper, “Retractions,” is published in the Review of Economics and Statistics, a peer-reviewed economics journal. The authors are Azoulay, the Sloan Distinguished Associate Professor of Management; Fiona Murray, the Alvin J. Siteman Professor of Entrepreneurship, associate dean for innovation at MIT Sloan, and co-director of MIT’s Innovation Initiative; Joshua Krieger, a doctoral student at MIT Sloan; and Jeffrey Furman PhD '02, an economist at Boston University.

Murray and Furman were co-authors of an influential 2012 paper that studied the circumstances in which retractions occur. The new paper, Azoulay notes, is different in that it spotlights “the consequences of retractions, not their antecedents.”

The current study focuses on the life sciences. The researchers examined the effects of more than 1,000 retractions of papers published between 1973 and 2007, and retracted by 2009. They also used the PubMed Related Citations Algorithm to help define the research fields relevant to those retracted papers.

Given those definitions of research fields, the researchers then examined the effect of retractions on intellectually related work. One group of 60,000 papers that they examined experienced the 6 percent decline in citation rates after retractions occurred in the same fields. To establish that fact, the researchers compared the citation trajectories of those papers with a control group of 110,000 papers that were published in the same journals. The data and method draw on work Azoulay has compiled and refined while developing numerous other analyses of citation rates in scientific literature.

But within these numbers lies another story: The citation rates for related papers that are still valid drop more precipitously when papers citing them are retracted for reasons of fraud or other misconduct, as opposed to, say, a laboratory mistake.

“Most of the declines in citations and funding we see are driven by fraud cases,” Krieger notes. “When an honest mistake happens, the related field doesn’t experience this big decline.”

Conversely, this means that, given two equally valid sets of past research, legitimate research cited in a paper later retracted for fraud will be more harshly punished.

Further evidence for the “stigma story,” as Krieger calls it, is that academic papers and nonacademic papers from firms tend to shun retracted papers at similar rates — but papers from commercial firms do not avoid citing older, still-valid related papers to the same extent as academic researchers. This could mean there is an unfounded flight away from those related papers in academic research.

“Our evidence pointing to the stigma story implies that funding agencies and investigators should be more cautious when deciding to abandon a field after a case of research misconduct,” Krieger adds. “We need to be careful in separating what we’ve learned about the field’s scientific status from our strong reactions to the disgrace of misconduct and fraud.”

Limits to knowledge

Other scholars in the field find the study valuable. Ben Jones, a professor at Northwestern University’s Kellogg School of Management, says the study “uses rigorous empirical methods to establish an important fact: Retractions matter for the ensuing progress of a field.” He adds: “There are good reasons to think that stigma effects may help explain these results.”

Jones, who was not involved in this research, was co-author of a 2013 paper indicating that authors of retracted papers avoid relatively larger citation losses when they self-report problems.

As Azoulay and his colleagues acknowledge, there are limits to their study. It’s possible that citations in fields affected by fraud logically should decline more than those marred by honest mistakes, because scientists may rightfully conclude that research areas containing outright fraud are further away from yielding productive results. 

“Maybe … fraud retractions and error retractions are different in ways that are correlated for the potential for follow-on research,” Azoulay says.

The researchers tried to account for that in their study, by looking at the number of citations garnered by retracted articles prior to the discovery of problems, then correlating that with the impact on related papers. But the numbers do not show an underlying structural reason why retractions for fraud, as opposed to unintentional mistakes, should generate a larger impact on related prior research.

The researchers have a couple of specific policy suggestions to limit unwarranted damage to the accretion of scientific knowledge. For instance, journals that retract articles should offer detailed explanations of why those papers are no longer valid. It would not be hard, they suggest, to develop a basic system of categories of retractions.

“By lumping together honest mistakes and misconduct, we’re undermining the smooth functioning of the retraction system,” Azoulay says.

Topics: Research, Social sciences, Sloan School of Management


Science is undeniably hard. First you come to grips with failure in the lab and when you finally believe you got it right, another front opens up: the quest to pass peer review and get your findings published in a good journal. These days, a new cloud looms over the embattled researcher right at the end of the publication pipeline: Post publication peer review (PPPR). PPPR has become the new way to challenge publications, at least that is what the self-published blog “Retraction Watch” by Adam Marcus and Ivan Oransky would like us to believe as they relentlessly report retractions and other sins and ruin reputations right and left. Calling PPPR peer review is of course a misnomer, since no real established peers are known to be involved, or enable their credentials to be checked, and no serious editorial participation of peers is involved at this post publication stage. PPPR is, for all we know and from what we have seen, a trigger-happy operation, conducted by a horde mostly hiding in anonymity and driven by anger and rage, seeking to get their easy share of fame.

In essence, the McCarthyian agenda of PPPR is drawn by journalists (it takes all kinds) and by angry people who, usually emboldended by anonymity, comment in Retraction Watch and other blogs, as they hysterically seek to find a place in history.

What motivates Marcus, Oransky and the post-publication transparency champions to embark in their retraction watchdog crusade? Not the quest for transparency, of course, otherwise they would play by the rules of science and demand that the post-publication challengers submit their comments to the incumbent journals seeking publication which would be granted provided, of course, that their comments successfully pass peer review. Transparency or fair play would then dictate that the author who has presumably sinned be given the chance to retort within the same forum and subject to the same rules of publication.

The motivation of the post-publication transparency champions is likely to be very different. Doing great science is hard, and yet there are easier routes to fame and one such route is to “bring down the phonies”, to paraphrase the novel “The Catcher in the Rye”. The frame of mind of the transparency champions is more akin to “I cannot do great science (or anything creative for that matter) so those who can surely must be phonies, right?” Wrong!

If this sounds familiar it is simply because it fits into an ancient psychological pattern (“sour grapes”), first illustrated by Aesop and probably as old as human civilization. It took a grotesque and tragic turn in the John Lennon assassination at the hands of Mark David Chapman, a criminal obsessed with “The Catcher in the Rye” and the “anti-phoniness” message.

Let us hope the science journals recover their lost ground and drive away the angry hordes as they regain the transparency territory. Scientific McCarthyism will only come to an end when the scientific establishment tightens up peer review.

Back to the top