If you won a billion dollars in the lottery, every distant aunt, uncle and cousin along with dozens of non-profit organizations and assorted others would come to you with their hands out. You’re a giving person, but how do you determine who is most worthy of receiving some money?
That’s the analogy Keri Cagle uses when talking about the value of peer review.
Cagle is the director of peer review for ORAU and the Oak Ridge Institute for Science and Education (ORISE). Her lottery analogy is simplistic but it helps people better understand the work she and her team do every day, she said.
She added that peer review and evaluation is the unsung hero of the research funding and proposal evaluation process, yet it is critically important to ensuring available research dollars are spent wisely.
“Peer review really does not get the credit it deserves because so much of the process happens behind the scenes,” she says. “Peer review is not the actual research being done. It’s an evaluation of research funding proposals, or research project proposals, from researchers who are requesting funding for their work.”
All federal- and state-funded research projects are mandated by law to be peer reviewed, Cagle said. ORAU has conducted peer reviews for a number of governmental agencies and other institutions, including the Department of Energy, Florida Department of Health, Pennsylvania Department of Health, Nazarbayev University in Kazakhstan, and more.
ORAU’s peer review team manages the entire life cycle of the peer review process, which includes the following:
- Facilitate and coordinate advisory committee meetings and workshops where decisions are made about which topics should be researched and funded. These advisory committees are usually comprised of agency-selected subject matter experts who determine what research priorities should be.
- Collect proposals from researchers through PeerNet, ORAU’s proprietary peer review management portal.
- Recruit and identify subject matter experts from ORAU’s database of more than 18,000 global experts on a wide array of topics.
- Determine whether the review meeting should be held in person or via webinar, then coordinate the execution of the meeting
- Provide results to the customer, who will use the information to make their funding decisions.
The entire process helps ensure our customers get the very best value for the research funding dollars they have available, Cagle said.
Throughout the process, Cagle and her team are focused on maintaining integrity.
“Integrity is number one. That means the products that we are putting together and the services we provide need to be of the highest quality,” she said. “If we’re not providing quality service, how can we justify the integrity of that review.”
A peer review conducted with integrity is a review that is free of conflicts of interest, and with minimal bias or strong opinions of peer reviewers.
“It’s difficult to control for bias, but we do monitor for conflict of interest, and we have processes in place to avoid it,” Cagle said.
Each agency has its own criteria for what constitutes conflict of interest, but typically a conflict of interest would exist if you are a subject matter expert who has had some form of involvement, be it a friendship or a working relationship, with the researcher or a member of the research team submitting the proposal.
“Integrity is important because as good stewards of research dollars, the last thing we want is for someone to come back and question how the funds were awarded,” Cagle said.
She added that one of the benefits of PeerNet is that there is an electronic record of interactions with reviewers, including questions about whether they have conflicts of interest with the members of a proposal team.
“It shows we’ve done our due diligence to ensure integrity throughout the process,” she said.
Additionally, one of her project managers sits in on panel discussions to monitor for conflicts of interest.
“I have personally stopped a review discussion because I knew someone in the room hadn’t been asked to leave, and I had to remind the panel chair that this person had a conflict of interest and needed to be out of the room during the discussion,” Cagle said.
None of this happens in a vacuum, of course. Cagle’s team includes research associates, project managers, project support teams, technology support teams and certified meeting planners.“It takes a village to make these reviews successful,” she said.