The Cutting Edge of Medical Technology Content, Community & Collaboration
Is Peer Review Broken?
Colleagues,
As published in GenomeWeb News please see excerpts from this provocative and thoughtful essay on the counter intuitive nature of using a "subjective" peer review system to judge the most important elements of an "objective" field such as science ...
-----------------------------------------------------------------------------------------------------------------------------
.... It is a fundamental irony of science that a field created for and by people who dedicate their lives to eschewing opinion in favor of measurable, objective standards relies as a gauge of success almost entirely on a subjective system: peer review. Scientists' career milestones — from publishing that pivotal paper to winning that crucial grant — hinge on human opinion...
... It's no secret that most scientists dislike the current peer review system — for a number of reasons, from how much time it takes to the seeming capriciousness of review decisions....
... In the pages that follow, we'll look into the peer review processes for grants and for papers, both of which have seen improvements and experimentation in recent years. While it may seem unwieldy to lump them together, concerns about the basic concept of peer review tend to underlie both systems, and it may be informative to consider them jointly.
Grant review
There was nothing quite like the US stimulus fund program to put the grant peer review process in the hot seat. For the National Institutes of Health, timing probably couldn't have been worse: the agency was in the throes of a series of changes to the review program set in motion by Elias Zerhouni before he stepped down from his role as director. In the midst of that, NIH received somewhere in the neighborhood of 30,000 grant applications for its bolus of new funding, which necessitated finding 20,000 extra reviewers and training them in the new process for a review that had to take place in record time. As far as a stress test of the system, you really couldn't ask for more...
... A general complaint about grant review is the need for more and higher-quality reviewers ... In general, (some believe) .... the grant review process is in need of a complete change, rather than minor adjustments. No company would be successful "if over half [its scientists'] time was spent trying to justify getting funded, and then you would only let one out of every 10 do any work," .... "Yet the country runs its R&D department that way."
Journal review
Whether you're a veteran scientist with dozens of published papers under your belt or a novice with just one or two publications to your name, chances are that you've experienced the feeling that the peer review process for scientific literature leaves something to be desired. The laundry list of things scientists wish would be improved in the process is quite lengthy and includes items such as more comprehensive reviews, reviewers who are more familiar with the science of the paper at hand, or just wanting a paper to get past an editor and out for review....
... What everyone can agree on is that trying new approaches with the journal review system can only help. "Experimenting with some of these alternatives is going to give us more options," ...
Themes
There are a number of themes that cut across journal and grant peer review. For instance, the rise of interdisciplinary science makes it less and less likely that any one or two people can have expertise in all aspects of a research proposal or a submitted paper...
... "The bottom line is, we should be thinking about how to make scientists as successful and productive as they could be," ... "A system where you spend half your time on fundraising is not a smart system."
---------------------------------------------------------------------------------------------------------------------------
Read the full essay at: http://www.genomeweb.com/peer-review-broken
ENJOY!
CC
As published in GenomeWeb News please see excerpts from this provocative and thoughtful essay on the counter intuitive nature of using a "subjective" peer review system to judge the most important elements of an "objective" field such as science ...
-----------------------------------------------------------------------------------------------------------------------------
.... It is a fundamental irony of science that a field created for and by people who dedicate their lives to eschewing opinion in favor of measurable, objective standards relies as a gauge of success almost entirely on a subjective system: peer review. Scientists' career milestones — from publishing that pivotal paper to winning that crucial grant — hinge on human opinion...
... It's no secret that most scientists dislike the current peer review system — for a number of reasons, from how much time it takes to the seeming capriciousness of review decisions....
... In the pages that follow, we'll look into the peer review processes for grants and for papers, both of which have seen improvements and experimentation in recent years. While it may seem unwieldy to lump them together, concerns about the basic concept of peer review tend to underlie both systems, and it may be informative to consider them jointly.
Grant review
There was nothing quite like the US stimulus fund program to put the grant peer review process in the hot seat. For the National Institutes of Health, timing probably couldn't have been worse: the agency was in the throes of a series of changes to the review program set in motion by Elias Zerhouni before he stepped down from his role as director. In the midst of that, NIH received somewhere in the neighborhood of 30,000 grant applications for its bolus of new funding, which necessitated finding 20,000 extra reviewers and training them in the new process for a review that had to take place in record time. As far as a stress test of the system, you really couldn't ask for more...
... A general complaint about grant review is the need for more and higher-quality reviewers ... In general, (some believe) .... the grant review process is in need of a complete change, rather than minor adjustments. No company would be successful "if over half [its scientists'] time was spent trying to justify getting funded, and then you would only let one out of every 10 do any work," .... "Yet the country runs its R&D department that way."
Journal review
Whether you're a veteran scientist with dozens of published papers under your belt or a novice with just one or two publications to your name, chances are that you've experienced the feeling that the peer review process for scientific literature leaves something to be desired. The laundry list of things scientists wish would be improved in the process is quite lengthy and includes items such as more comprehensive reviews, reviewers who are more familiar with the science of the paper at hand, or just wanting a paper to get past an editor and out for review....
... What everyone can agree on is that trying new approaches with the journal review system can only help. "Experimenting with some of these alternatives is going to give us more options," ...
Themes
There are a number of themes that cut across journal and grant peer review. For instance, the rise of interdisciplinary science makes it less and less likely that any one or two people can have expertise in all aspects of a research proposal or a submitted paper...
... "The bottom line is, we should be thinking about how to make scientists as successful and productive as they could be," ... "A system where you spend half your time on fundraising is not a smart system."
---------------------------------------------------------------------------------------------------------------------------
Read the full essay at: http://www.genomeweb.com/peer-review-broken
ENJOY!
CC
Views: 13
Comment
MedTechIQ VideoLibrary
Upcoming Events
© 2024 Created by CC-Conrad Clyburn-MedForeSight.
Powered by
You need to be a member of MedTech I.Q. to add comments!
Join MedTech I.Q.