Peer review is critical to science. Subjecting research proposals to evaluation by a panel of experts helps funding entities determine the scientific merit of proposals submitted for funding, helping these agencies ensure their research dollars are being spent wisely.
A peer review panel, though, may only be as strong as the skills of its member panelists, suggests a new study published in the research journal, PLOS ONE, on May 13, 2020. The study, titled, “What makes an effective peer reviewer? An exploratory study of the necessary skills,” was conducted by four current and former ORAU researchers. It takes a different approach than other studies of the peer review process, said Tiffani Conner, Ph.D., ORAU social scientist, and one of the study’s authors.
“This study is unique in the literature of peer review, most of which focuses on outcomes and monetary awards – looking only at numbers and counts,” Conner said. “This study really sets ORAU apart from other groups because of our focus on the skills and training that might be necessary for peer reviewers. Peer review literature from the last three or four years calls for more research from a social science perspective and our study is among the first of its kind.”
Conner explained that data were collected through interviews with seven program officers and five expert peer review panelists, and surveys from 51 respondents. While the results cannot be broadly generalized based on the number of responses, study authors gained insight into the skills required of peer reviewers. In total, 21 skills were identified, including subject matter expertise, familiarity with the peer review process, broad scientific knowledge, analytical thinking and impartiality.
Through their interviews, authors identified 13 activities that can improve peer reviewers’ skills, including observation of other panelists, writing/submitting research proposals, listening to other panelists make arguments, and mentoring/being mentored by other peer reviewers.
Conner said authors also considered two specific review panel formats: in-person face-to-face and virtual video conference.
“In general, in-person peer review panels were preferred over virtual video conference,” Conner said. “The perception is that communication and engagement is stronger during in-person meetings, but all of the skills are necessary regardless of format.”
She added that more research needs to be conducted on the skills necessary for effective peer review, and that this study is a great first step in that direction.
“ORAU is on the modern edge of the current and future considerations by funding agencies – that of multi-disciplinary and diverse review panels,” she said. “This article demonstrates ORAU’s thought leadership, demonstrates ORAU researchers’ skills as social scientists, and fulfills a notable outcome for the Oak Ridge Institute for Science and Education. ORISE is managed by ORAU for the U.S. Department of Energy.
Co-authors for the study include: Miriam Davis, research associate in ORAU Research Services; Kate Miller-Bains, evaluation specialist in ORAU Analysis and Evaluation; and Leslie Shapard, retired.