Scientists are much more likely to produce innovative research when using long-term grants that allow them exceptional freedom in the lab, according to a new study co-written by MIT economists.
The work shows that biologists whose funding encourages them to take risks and tolerates initial research failures wind up producing about twice as many highly influential papers as some peers whose funding is dependent upon meeting closely defined, short-term research targets.
“If you want people to branch out in new directions, then it’s important to provide for their long-term horizons, to give them time to experiment and potentially fail,” says Pierre Azoulay, an associate professor at the MIT Sloan School of Management, and an author of the study. “The researcher has to believe that short-term failure will not be punished.”
The results are contained in a working paper released this fall, “Incentives and Creativity: Evidence from the Academic Life Sciences,” by Azoulay, Gustavo Manso, an assistant professor at Sloan, and Joshua Graff Zivin, an associate professor of economics at the University of California, San Diego.
The researchers believe their evidence shows it is possible to manage lab work in a way that increases the chances that scientists will produce breakthrough findings, not just incremental advances within an established paradigm. “You can generate innovation, but the details matter,” says Azoulay. “What you want to provide incentives for is future performance, not performance today.”
The study appears as science funding has recently risen in the United States, in part through the stimulus bill Congress passed in 2009, which provided about $20 billion for research. Not counting stimulus money, President Barack Obama still included a slight increase in federal support for science as part of his proposed 2010 budget, which asks for about $148 billion for research and development. In April, Obama suggested that scientific funding should equal 3 percent of America’s economic production. Azoulay says he and his colleagues would like to instigate a discussion about not only how much money should be spent on research, but how those funds should be managed.
Measuring creativity
Azoulay, Manso, and Graff Zivin arrived at their conclusions after comparing researchers using two distinct types of funding: support from the investigator program of the Howard Hughes Medical Institute (HHMI), the large non-profit biomedical research organization in Maryland, and the R01 grants of the National Institutes of Health (NIH), the federal government’s life-science center in Maryland. The HHMI support lasts five years and is often renewed; the program “urges its researchers to take risks … even if it means uncertainty or the chance of failure.” The HHMI also provides a two-year buffer of support after funding is terminated. The NIH grants last three to five years, have more specific aims, and cease immediately if not renewed.
The researchers identified 73 life scientists given HHMI support in three years — 1993, 1994, and 1995 — and tracked their work through 2006. Because these scientists were quite well-regarded before getting HHMI funding, the study compared them to groups of similarly accomplished scientists receiving NIH grants: one group of 393 scientists who had received early-career prizes, and another group of 92 scientists receiving the NIH’s MERIT funding, awarded to highly promising projects.
Among other things, Azoulay, Manso, and Graff Zivin analyzed how often these scientists published articles that were among the top 5 percent or top 1 percent of the most cited papers in their fields. They also studied “creativity” in lab research by seeing how often the scientists began using new keywords to describe the subjects of their articles.
Their findings show that compared to the early-career prize winners with NIH grants, the HHMI-funded scientists produced twice as many papers in the top 5 percent in terms of citations, and three times as many in the top 1 percent. Compared to the NIH-funded scientists with MERIT grants, the HHMI group produced about the same quantity of papers in the top 5 percent by citation, but 50 percent more papers in the top 1 percent.
The study also found that the HHMI investigators had about 10 percent more variety in the keywords they introduced into their own work than the early-career prizewinners from the NIH, and were cited in a greater range of journals. Additionally, the HHMI-backed scientists mentored more early-career prize-winning scientists themselves (1.13 per person) compared to the NIH-funded group (0.24 per person).
Avice Meehan, vice president for comunications and public affairs at HHMI, says the study reflects the fact that over the last two decades, “HHMI has identified highly creative scientists and given them the freedom to pursue critical medical research, even if it takes them years, and means a change of research direction.”
The view from the NIH
The researchers acknowledge that measures such as keywords are imperfect indicators of creativity, but think such tools are a reasonable way of identifying originality in the lab. “There are as many definitions of creativity as there are people studying creativity,” acknowledges Azoulay. “But ultimately creativity is measured in especially good outcomes.”
Azoulay, Manso, and Graff Zivin also emphasize that their work is not an institutional critique of the NIH. “The conclusion of our paper is not that the NIH should transform itself into a version of the HHMI,” Azoulay adds. Their larger point simply concerns the effects of different types of grants. If major discoveries are not unanticipated events, but influenced by the underlying funding, policy-makers could consider that point when allocating research dollars.
Moreover, the civic value of science often comes not only from an initial breakthrough, but later incremental refinements of it. In those cases, shorter-term, narrower research provides significant social benefits. “It’s an outstanding question what the actual mix of exploration and exploitation we need is,” Azoulay notes.
Don Ralbovsky, an NIH spokesperson, said a staff member in the NIH’s Office of Extramural Research had looked at the paper and described it as “interesting,” but would refrain from further comment until the paper appears in final published form.
In recent years, the NIH has developed multiple types of funding beyond the traditional R01 grants. The Pioneer Award, founded in 2004, is a grant for “highly innovative new research approaches,” to be given to seven scientists in 2010. The New Innovator Award is for 33 early-career investigators in 2010, emphasizing “innovation and potential impact.” And in 2008, the NIH established Transformative Research projects Awards, making $25 million available for “bold and creative investigator–initiated research.” All of these grants last five years, instead of three for the standard R01 grants.
Azoulay agrees that the existence of a variety of types of grants can help science as a whole. “A division of labor might benefit of the entire research ecosystem,” he says. The HHMI’s Meehan concurs: “It’s important for the nation to have a comprehensive research portfolio that encompasses many approaches and mechanisms.” (This research was funded in part by the Kauffman Foundation and the Science of Science Policy Program of the National Science Foundation.)
One long-term goal of Azoulay’s work is “to bring randomized trials to science policy.” By comparing two groups over time, this study attempts to replicate the lab-trial method, albeit with historical data, and shed more empirical light on a subject often discussed anecdotally.
“This is the first word on the topic, not the last,” concludes Azoulay.
The work shows that biologists whose funding encourages them to take risks and tolerates initial research failures wind up producing about twice as many highly influential papers as some peers whose funding is dependent upon meeting closely defined, short-term research targets.
“If you want people to branch out in new directions, then it’s important to provide for their long-term horizons, to give them time to experiment and potentially fail,” says Pierre Azoulay, an associate professor at the MIT Sloan School of Management, and an author of the study. “The researcher has to believe that short-term failure will not be punished.”
The results are contained in a working paper released this fall, “Incentives and Creativity: Evidence from the Academic Life Sciences,” by Azoulay, Gustavo Manso, an assistant professor at Sloan, and Joshua Graff Zivin, an associate professor of economics at the University of California, San Diego.
The researchers believe their evidence shows it is possible to manage lab work in a way that increases the chances that scientists will produce breakthrough findings, not just incremental advances within an established paradigm. “You can generate innovation, but the details matter,” says Azoulay. “What you want to provide incentives for is future performance, not performance today.”
The study appears as science funding has recently risen in the United States, in part through the stimulus bill Congress passed in 2009, which provided about $20 billion for research. Not counting stimulus money, President Barack Obama still included a slight increase in federal support for science as part of his proposed 2010 budget, which asks for about $148 billion for research and development. In April, Obama suggested that scientific funding should equal 3 percent of America’s economic production. Azoulay says he and his colleagues would like to instigate a discussion about not only how much money should be spent on research, but how those funds should be managed.
Measuring creativity
Azoulay, Manso, and Graff Zivin arrived at their conclusions after comparing researchers using two distinct types of funding: support from the investigator program of the Howard Hughes Medical Institute (HHMI), the large non-profit biomedical research organization in Maryland, and the R01 grants of the National Institutes of Health (NIH), the federal government’s life-science center in Maryland. The HHMI support lasts five years and is often renewed; the program “urges its researchers to take risks … even if it means uncertainty or the chance of failure.” The HHMI also provides a two-year buffer of support after funding is terminated. The NIH grants last three to five years, have more specific aims, and cease immediately if not renewed.
The researchers identified 73 life scientists given HHMI support in three years — 1993, 1994, and 1995 — and tracked their work through 2006. Because these scientists were quite well-regarded before getting HHMI funding, the study compared them to groups of similarly accomplished scientists receiving NIH grants: one group of 393 scientists who had received early-career prizes, and another group of 92 scientists receiving the NIH’s MERIT funding, awarded to highly promising projects.
Among other things, Azoulay, Manso, and Graff Zivin analyzed how often these scientists published articles that were among the top 5 percent or top 1 percent of the most cited papers in their fields. They also studied “creativity” in lab research by seeing how often the scientists began using new keywords to describe the subjects of their articles.
Their findings show that compared to the early-career prize winners with NIH grants, the HHMI-funded scientists produced twice as many papers in the top 5 percent in terms of citations, and three times as many in the top 1 percent. Compared to the NIH-funded scientists with MERIT grants, the HHMI group produced about the same quantity of papers in the top 5 percent by citation, but 50 percent more papers in the top 1 percent.
The study also found that the HHMI investigators had about 10 percent more variety in the keywords they introduced into their own work than the early-career prizewinners from the NIH, and were cited in a greater range of journals. Additionally, the HHMI-backed scientists mentored more early-career prize-winning scientists themselves (1.13 per person) compared to the NIH-funded group (0.24 per person).
Avice Meehan, vice president for comunications and public affairs at HHMI, says the study reflects the fact that over the last two decades, “HHMI has identified highly creative scientists and given them the freedom to pursue critical medical research, even if it takes them years, and means a change of research direction.”
The view from the NIH
The researchers acknowledge that measures such as keywords are imperfect indicators of creativity, but think such tools are a reasonable way of identifying originality in the lab. “There are as many definitions of creativity as there are people studying creativity,” acknowledges Azoulay. “But ultimately creativity is measured in especially good outcomes.”
Azoulay, Manso, and Graff Zivin also emphasize that their work is not an institutional critique of the NIH. “The conclusion of our paper is not that the NIH should transform itself into a version of the HHMI,” Azoulay adds. Their larger point simply concerns the effects of different types of grants. If major discoveries are not unanticipated events, but influenced by the underlying funding, policy-makers could consider that point when allocating research dollars.
Moreover, the civic value of science often comes not only from an initial breakthrough, but later incremental refinements of it. In those cases, shorter-term, narrower research provides significant social benefits. “It’s an outstanding question what the actual mix of exploration and exploitation we need is,” Azoulay notes.
Don Ralbovsky, an NIH spokesperson, said a staff member in the NIH’s Office of Extramural Research had looked at the paper and described it as “interesting,” but would refrain from further comment until the paper appears in final published form.
In recent years, the NIH has developed multiple types of funding beyond the traditional R01 grants. The Pioneer Award, founded in 2004, is a grant for “highly innovative new research approaches,” to be given to seven scientists in 2010. The New Innovator Award is for 33 early-career investigators in 2010, emphasizing “innovation and potential impact.” And in 2008, the NIH established Transformative Research projects Awards, making $25 million available for “bold and creative investigator–initiated research.” All of these grants last five years, instead of three for the standard R01 grants.
Azoulay agrees that the existence of a variety of types of grants can help science as a whole. “A division of labor might benefit of the entire research ecosystem,” he says. The HHMI’s Meehan concurs: “It’s important for the nation to have a comprehensive research portfolio that encompasses many approaches and mechanisms.” (This research was funded in part by the Kauffman Foundation and the Science of Science Policy Program of the National Science Foundation.)
One long-term goal of Azoulay’s work is “to bring randomized trials to science policy.” By comparing two groups over time, this study attempts to replicate the lab-trial method, albeit with historical data, and shed more empirical light on a subject often discussed anecdotally.
“This is the first word on the topic, not the last,” concludes Azoulay.
